Jul 24 03:15:26.335611 ip-10-0-134-147 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 24 03:15:26.335623 ip-10-0-134-147 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 24 03:15:26.335630 ip-10-0-134-147 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 24 03:15:26.335942 ip-10-0-134-147 systemd[1]: Failed to start Kubernetes Kubelet. Jul 24 03:15:36.534550 ip-10-0-134-147 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 24 03:15:36.534568 ip-10-0-134-147 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 7040852bc0a84204a09f9e2273fe4d93 -- Jul 24 03:17:30.456825 ip-10-0-134-147 systemd[1]: Starting Kubernetes Kubelet... Jul 24 03:17:30.799359 ip-10-0-134-147 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 03:17:30.799359 ip-10-0-134-147 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 24 03:17:30.799359 ip-10-0-134-147 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 03:17:30.799359 ip-10-0-134-147 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 24 03:17:30.799359 ip-10-0-134-147 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 03:17:30.801450 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.801334 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 24 03:17:30.806129 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806107 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 03:17:30.806129 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806125 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 03:17:30.806129 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806129 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 03:17:30.806129 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806134 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 03:17:30.806129 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806137 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 03:17:30.806331 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806140 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 03:17:30.806331 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806143 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 03:17:30.806331 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806146 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 03:17:30.806331 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806149 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 03:17:30.806331 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806151 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 03:17:30.806331 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806154 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 03:17:30.806331 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806156 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 03:17:30.806331 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806159 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 03:17:30.806331 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806161 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 03:17:30.806331 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806164 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 03:17:30.806331 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806167 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 03:17:30.806331 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806169 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 03:17:30.806331 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806171 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 03:17:30.806331 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806174 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 03:17:30.806331 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806176 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 03:17:30.806331 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806179 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 03:17:30.806331 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806182 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 03:17:30.806331 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806184 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 03:17:30.806331 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806186 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 03:17:30.806331 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806189 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 03:17:30.806826 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806191 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 03:17:30.806826 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806194 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 03:17:30.806826 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806196 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 03:17:30.806826 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806198 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 03:17:30.806826 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806201 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 03:17:30.806826 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806203 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 03:17:30.806826 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806206 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 03:17:30.806826 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806208 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 03:17:30.806826 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806211 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 03:17:30.806826 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806213 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 03:17:30.806826 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806226 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 03:17:30.806826 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806229 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 03:17:30.806826 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806233 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 03:17:30.806826 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806236 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 03:17:30.806826 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806239 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 03:17:30.806826 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806242 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 03:17:30.806826 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806245 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 03:17:30.806826 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806247 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 03:17:30.806826 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806250 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 03:17:30.806826 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806252 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 03:17:30.807312 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806255 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 24 03:17:30.807312 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806257 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 03:17:30.807312 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806259 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 03:17:30.807312 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806262 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 03:17:30.807312 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806264 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 03:17:30.807312 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806267 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 03:17:30.807312 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806269 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 03:17:30.807312 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806272 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 03:17:30.807312 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806275 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 03:17:30.807312 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806279 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 03:17:30.807312 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806281 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 03:17:30.807312 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806283 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 03:17:30.807312 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806286 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 03:17:30.807312 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806288 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 03:17:30.807312 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806291 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 03:17:30.807312 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806293 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 03:17:30.807312 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806296 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 03:17:30.807312 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806298 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 03:17:30.807312 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806300 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 03:17:30.807777 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806303 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 03:17:30.807777 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806305 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 03:17:30.807777 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806307 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 03:17:30.807777 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806310 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 03:17:30.807777 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806312 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 03:17:30.807777 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806315 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 03:17:30.807777 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806318 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 03:17:30.807777 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806321 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 03:17:30.807777 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806323 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 03:17:30.807777 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806326 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 03:17:30.807777 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806328 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 03:17:30.807777 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806331 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 03:17:30.807777 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806333 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 03:17:30.807777 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806337 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 03:17:30.807777 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806342 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 03:17:30.807777 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806346 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 03:17:30.807777 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806350 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 03:17:30.807777 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806353 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 03:17:30.807777 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806356 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 03:17:30.808258 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806360 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 03:17:30.808258 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806362 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 03:17:30.808258 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806365 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 03:17:30.808258 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806367 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 03:17:30.808258 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806370 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 03:17:30.808258 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806373 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 03:17:30.808258 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806375 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 03:17:30.808258 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806378 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 03:17:30.808258 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806380 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 03:17:30.808258 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806384 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 03:17:30.808258 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806387 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 03:17:30.808258 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806391 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 03:17:30.808258 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806394 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 03:17:30.808258 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806396 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 03:17:30.808258 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806399 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 03:17:30.808258 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.806401 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 03:17:30.808258 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807138 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 03:17:30.808258 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807145 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 03:17:30.808258 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807149 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 03:17:30.808258 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807152 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 03:17:30.808761 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807155 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 03:17:30.808761 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807157 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 03:17:30.808761 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807160 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 03:17:30.808761 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807162 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 03:17:30.808761 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807166 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 03:17:30.808761 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807168 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 03:17:30.808761 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807171 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 24 03:17:30.808761 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807173 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 03:17:30.808761 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807175 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 03:17:30.808761 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807178 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 03:17:30.808761 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807181 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 03:17:30.808761 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807183 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 03:17:30.808761 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807185 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 03:17:30.808761 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807188 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 03:17:30.808761 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807191 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 03:17:30.808761 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807193 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 03:17:30.808761 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807196 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 03:17:30.808761 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807198 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 03:17:30.808761 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807201 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 03:17:30.808761 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807203 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 03:17:30.809263 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807206 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 03:17:30.809263 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807208 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 03:17:30.809263 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807210 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 03:17:30.809263 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807213 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 03:17:30.809263 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807215 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 03:17:30.809263 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807218 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 03:17:30.809263 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807220 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 03:17:30.809263 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807222 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 03:17:30.809263 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807225 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 03:17:30.809263 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807227 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 03:17:30.809263 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807230 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 03:17:30.809263 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807233 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 03:17:30.809263 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807236 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 03:17:30.809263 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807239 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 03:17:30.809263 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807241 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 03:17:30.809263 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807244 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 03:17:30.809263 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807246 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 03:17:30.809263 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807249 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 03:17:30.809263 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807251 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 03:17:30.809263 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807254 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 03:17:30.809785 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807256 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 03:17:30.809785 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807259 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 03:17:30.809785 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807261 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 03:17:30.809785 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807264 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 03:17:30.809785 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807266 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 03:17:30.809785 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807268 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 03:17:30.809785 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807272 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 03:17:30.809785 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807275 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 03:17:30.809785 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807277 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 03:17:30.809785 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807279 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 03:17:30.809785 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807282 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 03:17:30.809785 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807284 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 03:17:30.809785 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807287 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 03:17:30.809785 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807291 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 03:17:30.809785 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807294 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 03:17:30.809785 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807297 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 03:17:30.809785 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807300 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 03:17:30.809785 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807302 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 03:17:30.809785 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807305 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 03:17:30.810251 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807308 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 03:17:30.810251 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807310 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 03:17:30.810251 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807315 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 03:17:30.810251 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807320 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 03:17:30.810251 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807324 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 03:17:30.810251 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807327 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 03:17:30.810251 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807330 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 03:17:30.810251 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807332 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 03:17:30.810251 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807335 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 03:17:30.810251 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807337 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 03:17:30.810251 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807340 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 03:17:30.810251 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807342 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 03:17:30.810251 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807345 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 03:17:30.810251 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807347 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 03:17:30.810251 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807350 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 03:17:30.810251 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807352 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 03:17:30.810251 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807354 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 03:17:30.810251 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807357 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 03:17:30.810251 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807361 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 03:17:30.810735 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807364 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 03:17:30.810735 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807367 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 03:17:30.810735 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807370 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 03:17:30.810735 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807372 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 03:17:30.810735 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807375 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 03:17:30.810735 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807378 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 03:17:30.810735 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807380 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 03:17:30.810735 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807382 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 03:17:30.810735 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807387 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 03:17:30.810735 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807390 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 03:17:30.810735 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807393 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 03:17:30.810735 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807395 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 03:17:30.810735 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807399 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 03:17:30.810735 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807402 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 03:17:30.810735 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807404 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 03:17:30.810735 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807407 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 03:17:30.810735 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807409 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 03:17:30.810735 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807796 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 03:17:30.810735 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807802 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 03:17:30.810735 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807804 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 03:17:30.811234 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807807 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 03:17:30.811234 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807810 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 03:17:30.811234 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807813 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 03:17:30.811234 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807816 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 03:17:30.811234 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807818 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 03:17:30.811234 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807821 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 03:17:30.811234 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807824 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 03:17:30.811234 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807826 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 03:17:30.811234 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807829 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 03:17:30.811234 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807831 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 03:17:30.811234 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807833 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 03:17:30.811234 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807837 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 03:17:30.811234 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807840 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 03:17:30.811234 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807844 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 03:17:30.811234 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807846 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 03:17:30.811234 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807849 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 03:17:30.811234 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807851 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 03:17:30.811234 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807854 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 03:17:30.811234 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807856 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 03:17:30.811698 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807859 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 03:17:30.811698 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807862 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 03:17:30.811698 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807864 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 03:17:30.811698 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807867 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 03:17:30.811698 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807869 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 03:17:30.811698 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807872 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 03:17:30.811698 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807874 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 03:17:30.811698 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807876 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 03:17:30.811698 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807879 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 03:17:30.811698 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807881 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 03:17:30.811698 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807884 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 03:17:30.811698 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807887 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 03:17:30.811698 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807889 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 03:17:30.811698 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807892 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 03:17:30.811698 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807894 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 03:17:30.811698 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807897 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 03:17:30.811698 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807899 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 03:17:30.811698 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807902 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 03:17:30.811698 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807904 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 03:17:30.811698 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807907 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 03:17:30.812193 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807909 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 03:17:30.812193 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807911 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 03:17:30.812193 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807914 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 03:17:30.812193 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807916 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 03:17:30.812193 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807919 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 03:17:30.812193 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807921 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 03:17:30.812193 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807923 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 03:17:30.812193 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807927 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 03:17:30.812193 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807931 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 03:17:30.812193 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807933 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 03:17:30.812193 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807936 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 03:17:30.812193 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807938 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 03:17:30.812193 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807941 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 03:17:30.812193 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807943 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 03:17:30.812193 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807946 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 03:17:30.812193 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807948 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 03:17:30.812193 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807950 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 03:17:30.812193 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807953 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 03:17:30.812193 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807955 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 03:17:30.812193 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807958 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 03:17:30.812753 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807961 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 03:17:30.812753 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807963 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 03:17:30.812753 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807966 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 03:17:30.812753 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807969 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 03:17:30.812753 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807972 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 03:17:30.812753 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807974 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 03:17:30.812753 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807977 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 03:17:30.812753 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807980 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 03:17:30.812753 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807982 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 03:17:30.812753 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807985 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 03:17:30.812753 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807987 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 03:17:30.812753 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807989 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 03:17:30.812753 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807992 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 03:17:30.812753 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807994 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 03:17:30.812753 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.807997 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 03:17:30.812753 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808000 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 03:17:30.812753 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808002 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 03:17:30.812753 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808004 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 03:17:30.812753 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808007 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 03:17:30.812753 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808009 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 03:17:30.813246 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808013 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 24 03:17:30.813246 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808017 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 03:17:30.813246 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808020 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 03:17:30.813246 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808022 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 03:17:30.813246 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808025 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 03:17:30.813246 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808027 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 03:17:30.813246 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808030 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 03:17:30.813246 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808032 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 03:17:30.813246 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808036 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 03:17:30.813246 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808039 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 03:17:30.813246 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808042 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 03:17:30.813246 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808045 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 03:17:30.813246 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808047 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 03:17:30.813246 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808050 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 03:17:30.813246 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808053 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 03:17:30.813246 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808055 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 03:17:30.813246 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808058 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 03:17:30.813246 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808141 2579 flags.go:64] FLAG: --address="0.0.0.0" Jul 24 03:17:30.813246 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808151 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 24 03:17:30.813246 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808162 2579 flags.go:64] FLAG: --anonymous-auth="true" Jul 24 03:17:30.813246 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808168 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 24 03:17:30.813768 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808173 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 24 03:17:30.813768 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808176 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 24 03:17:30.813768 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808180 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 24 03:17:30.813768 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808184 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 24 03:17:30.813768 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808187 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 24 03:17:30.813768 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808190 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 24 03:17:30.813768 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808193 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 24 03:17:30.813768 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808196 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 24 03:17:30.813768 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808199 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 24 03:17:30.813768 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808202 2579 flags.go:64] FLAG: --cgroup-root="" Jul 24 03:17:30.813768 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808205 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 24 03:17:30.813768 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808208 2579 flags.go:64] FLAG: --client-ca-file="" Jul 24 03:17:30.813768 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808213 2579 flags.go:64] FLAG: --cloud-provider="external" Jul 24 03:17:30.813768 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808216 2579 flags.go:64] FLAG: --cluster-dns="[]" Jul 24 03:17:30.813768 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808220 2579 flags.go:64] FLAG: --cluster-domain="" Jul 24 03:17:30.813768 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808223 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 24 03:17:30.813768 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808226 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 24 03:17:30.813768 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808228 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 24 03:17:30.813768 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808232 2579 flags.go:64] FLAG: --container-log-max-files="5" Jul 24 03:17:30.813768 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808235 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 24 03:17:30.813768 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808238 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 24 03:17:30.813768 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808241 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 24 03:17:30.813768 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808244 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 24 03:17:30.814322 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808247 2579 flags.go:64] FLAG: --contention-profiling="false" Jul 24 03:17:30.814322 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808250 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 24 03:17:30.814322 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808252 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 24 03:17:30.814322 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808255 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 24 03:17:30.814322 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808258 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 24 03:17:30.814322 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808263 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 24 03:17:30.814322 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808266 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 24 03:17:30.814322 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808269 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 24 03:17:30.814322 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808272 2579 flags.go:64] FLAG: --enable-load-reader="false" Jul 24 03:17:30.814322 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808275 2579 flags.go:64] FLAG: --enable-server="true" Jul 24 03:17:30.814322 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808277 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 24 03:17:30.814322 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808282 2579 flags.go:64] FLAG: --event-burst="100" Jul 24 03:17:30.814322 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808285 2579 flags.go:64] FLAG: --event-qps="50" Jul 24 03:17:30.814322 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808288 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 24 03:17:30.814322 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808291 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 24 03:17:30.814322 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808294 2579 flags.go:64] FLAG: --eviction-hard="" Jul 24 03:17:30.814322 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808297 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 24 03:17:30.814322 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808301 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 24 03:17:30.814322 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808304 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 24 03:17:30.814322 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808307 2579 flags.go:64] FLAG: --eviction-soft="" Jul 24 03:17:30.814322 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808310 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 24 03:17:30.814322 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808314 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 24 03:17:30.814322 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808317 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 24 03:17:30.814322 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808320 2579 flags.go:64] FLAG: --experimental-mounter-path="" Jul 24 03:17:30.814322 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808323 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 24 03:17:30.814958 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808325 2579 flags.go:64] FLAG: --fail-swap-on="true" Jul 24 03:17:30.814958 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808328 2579 flags.go:64] FLAG: --feature-gates="" Jul 24 03:17:30.814958 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808332 2579 flags.go:64] FLAG: --file-check-frequency="20s" Jul 24 03:17:30.814958 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808335 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 24 03:17:30.814958 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808337 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 24 03:17:30.814958 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808340 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 24 03:17:30.814958 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808343 2579 flags.go:64] FLAG: --healthz-port="10248" Jul 24 03:17:30.814958 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808346 2579 flags.go:64] FLAG: --help="false" Jul 24 03:17:30.814958 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808349 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-134-147.ec2.internal" Jul 24 03:17:30.814958 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808352 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 24 03:17:30.814958 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808355 2579 flags.go:64] FLAG: --http-check-frequency="20s" Jul 24 03:17:30.814958 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808357 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 24 03:17:30.814958 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808361 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 24 03:17:30.814958 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808364 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 24 03:17:30.814958 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808367 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 24 03:17:30.814958 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808370 2579 flags.go:64] FLAG: --image-service-endpoint="" Jul 24 03:17:30.814958 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808372 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 24 03:17:30.814958 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808375 2579 flags.go:64] FLAG: --kube-api-burst="100" Jul 24 03:17:30.814958 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808378 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 24 03:17:30.814958 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808381 2579 flags.go:64] FLAG: --kube-api-qps="50" Jul 24 03:17:30.814958 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808383 2579 flags.go:64] FLAG: --kube-reserved="" Jul 24 03:17:30.814958 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808386 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 24 03:17:30.814958 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808389 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 24 03:17:30.814958 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808392 2579 flags.go:64] FLAG: --kubelet-cgroups="" Jul 24 03:17:30.815589 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808396 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 24 03:17:30.815589 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808398 2579 flags.go:64] FLAG: --lock-file="" Jul 24 03:17:30.815589 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808401 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 24 03:17:30.815589 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808403 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 24 03:17:30.815589 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808407 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 24 03:17:30.815589 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808412 2579 flags.go:64] FLAG: --log-json-split-stream="false" Jul 24 03:17:30.815589 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808416 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 24 03:17:30.815589 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808418 2579 flags.go:64] FLAG: --log-text-split-stream="false" Jul 24 03:17:30.815589 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808421 2579 flags.go:64] FLAG: --logging-format="text" Jul 24 03:17:30.815589 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808424 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 24 03:17:30.815589 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808427 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 24 03:17:30.815589 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808430 2579 flags.go:64] FLAG: --manifest-url="" Jul 24 03:17:30.815589 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808432 2579 flags.go:64] FLAG: --manifest-url-header="" Jul 24 03:17:30.815589 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808437 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 24 03:17:30.815589 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808440 2579 flags.go:64] FLAG: --max-open-files="1000000" Jul 24 03:17:30.815589 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808444 2579 flags.go:64] FLAG: --max-pods="110" Jul 24 03:17:30.815589 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808446 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 24 03:17:30.815589 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808449 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 24 03:17:30.815589 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808452 2579 flags.go:64] FLAG: --memory-manager-policy="None" Jul 24 03:17:30.815589 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808455 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 24 03:17:30.815589 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808470 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 24 03:17:30.815589 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808473 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 24 03:17:30.815589 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808476 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 24 03:17:30.815589 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808483 2579 flags.go:64] FLAG: --node-status-max-images="50" Jul 24 03:17:30.815589 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808486 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 24 03:17:30.816233 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808489 2579 flags.go:64] FLAG: --oom-score-adj="-999" Jul 24 03:17:30.816233 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808491 2579 flags.go:64] FLAG: --pod-cidr="" Jul 24 03:17:30.816233 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808494 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 24 03:17:30.816233 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808499 2579 flags.go:64] FLAG: --pod-manifest-path="" Jul 24 03:17:30.816233 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808502 2579 flags.go:64] FLAG: --pod-max-pids="-1" Jul 24 03:17:30.816233 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808505 2579 flags.go:64] FLAG: --pods-per-core="0" Jul 24 03:17:30.816233 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808508 2579 flags.go:64] FLAG: --port="10250" Jul 24 03:17:30.816233 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808512 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 24 03:17:30.816233 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808515 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0dd4b4d5c12451203" Jul 24 03:17:30.816233 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808518 2579 flags.go:64] FLAG: --qos-reserved="" Jul 24 03:17:30.816233 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808521 2579 flags.go:64] FLAG: --read-only-port="10255" Jul 24 03:17:30.816233 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808523 2579 flags.go:64] FLAG: --register-node="true" Jul 24 03:17:30.816233 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808527 2579 flags.go:64] FLAG: --register-with-taints="" Jul 24 03:17:30.816233 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808531 2579 flags.go:64] FLAG: --registry-burst="10" Jul 24 03:17:30.816233 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808534 2579 flags.go:64] FLAG: --registry-qps="5" Jul 24 03:17:30.816233 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808537 2579 flags.go:64] FLAG: --reserved-cpus="" Jul 24 03:17:30.816233 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808540 2579 flags.go:64] FLAG: --reserved-memory="" Jul 24 03:17:30.816233 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808543 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 24 03:17:30.816233 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808546 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 24 03:17:30.816233 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808549 2579 flags.go:64] FLAG: --rotate-certificates="false" Jul 24 03:17:30.816233 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808552 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 24 03:17:30.816233 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808554 2579 flags.go:64] FLAG: --runonce="false" Jul 24 03:17:30.816233 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808557 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 24 03:17:30.816233 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808561 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 24 03:17:30.816233 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808563 2579 flags.go:64] FLAG: --seccomp-default="false" Jul 24 03:17:30.816882 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808566 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 24 03:17:30.816882 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808569 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 24 03:17:30.816882 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808572 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 24 03:17:30.816882 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808575 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 24 03:17:30.816882 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808578 2579 flags.go:64] FLAG: --storage-driver-password="root" Jul 24 03:17:30.816882 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808581 2579 flags.go:64] FLAG: --storage-driver-secure="false" Jul 24 03:17:30.816882 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808583 2579 flags.go:64] FLAG: --storage-driver-table="stats" Jul 24 03:17:30.816882 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808586 2579 flags.go:64] FLAG: --storage-driver-user="root" Jul 24 03:17:30.816882 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808589 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 24 03:17:30.816882 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808592 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 24 03:17:30.816882 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808594 2579 flags.go:64] FLAG: --system-cgroups="" Jul 24 03:17:30.816882 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808597 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 24 03:17:30.816882 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808602 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 24 03:17:30.816882 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808605 2579 flags.go:64] FLAG: --tls-cert-file="" Jul 24 03:17:30.816882 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808609 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 24 03:17:30.816882 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808612 2579 flags.go:64] FLAG: --tls-min-version="" Jul 24 03:17:30.816882 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808615 2579 flags.go:64] FLAG: --tls-private-key-file="" Jul 24 03:17:30.816882 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808618 2579 flags.go:64] FLAG: --topology-manager-policy="none" Jul 24 03:17:30.816882 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808620 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 24 03:17:30.816882 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808624 2579 flags.go:64] FLAG: --topology-manager-scope="container" Jul 24 03:17:30.816882 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808627 2579 flags.go:64] FLAG: --v="2" Jul 24 03:17:30.816882 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808631 2579 flags.go:64] FLAG: --version="false" Jul 24 03:17:30.816882 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808638 2579 flags.go:64] FLAG: --vmodule="" Jul 24 03:17:30.816882 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808642 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 24 03:17:30.816882 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.808645 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 24 03:17:30.817608 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808757 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 03:17:30.817608 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808760 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 03:17:30.817608 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808764 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 03:17:30.817608 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808767 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 03:17:30.817608 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808770 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 03:17:30.817608 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808773 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 03:17:30.817608 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808777 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 03:17:30.817608 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808779 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 03:17:30.817608 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808782 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 03:17:30.817608 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808785 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 03:17:30.817608 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808787 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 03:17:30.817608 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808790 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 03:17:30.817608 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808792 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 03:17:30.817608 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808795 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 24 03:17:30.817608 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808797 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 03:17:30.817608 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808800 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 03:17:30.817608 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808803 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 03:17:30.817608 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808805 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 03:17:30.817608 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808807 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 03:17:30.817608 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808810 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 03:17:30.818092 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808812 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 03:17:30.818092 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808817 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 03:17:30.818092 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808819 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 03:17:30.818092 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808822 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 03:17:30.818092 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808824 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 03:17:30.818092 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808828 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 03:17:30.818092 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808833 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 03:17:30.818092 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808836 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 03:17:30.818092 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808838 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 03:17:30.818092 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808841 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 03:17:30.818092 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808844 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 03:17:30.818092 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808846 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 03:17:30.818092 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808849 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 03:17:30.818092 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808852 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 03:17:30.818092 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808854 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 03:17:30.818092 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808857 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 03:17:30.818092 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808859 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 03:17:30.818092 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808861 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 03:17:30.818092 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808864 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 03:17:30.818588 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808866 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 03:17:30.818588 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808869 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 03:17:30.818588 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808871 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 03:17:30.818588 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808874 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 03:17:30.818588 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808876 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 03:17:30.818588 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808879 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 03:17:30.818588 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808881 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 03:17:30.818588 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808884 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 03:17:30.818588 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808886 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 03:17:30.818588 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808889 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 03:17:30.818588 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808891 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 03:17:30.818588 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808893 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 03:17:30.818588 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808896 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 03:17:30.818588 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808898 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 03:17:30.818588 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808902 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 03:17:30.818588 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808904 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 03:17:30.818588 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808907 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 03:17:30.818588 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808910 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 03:17:30.818588 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808912 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 03:17:30.818588 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808916 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 03:17:30.819077 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808918 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 03:17:30.819077 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808921 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 03:17:30.819077 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808923 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 03:17:30.819077 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808926 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 03:17:30.819077 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808929 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 03:17:30.819077 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808931 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 03:17:30.819077 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808934 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 03:17:30.819077 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808936 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 03:17:30.819077 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808939 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 03:17:30.819077 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808942 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 03:17:30.819077 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808945 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 03:17:30.819077 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808948 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 03:17:30.819077 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808951 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 03:17:30.819077 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808953 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 03:17:30.819077 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808956 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 03:17:30.819077 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808959 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 03:17:30.819077 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808961 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 03:17:30.819077 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808963 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 03:17:30.819077 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808966 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 03:17:30.819077 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808968 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 03:17:30.819587 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808970 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 03:17:30.819587 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808973 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 03:17:30.819587 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808975 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 03:17:30.819587 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808978 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 03:17:30.819587 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808980 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 03:17:30.819587 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808982 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 03:17:30.819587 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808987 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 03:17:30.819587 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808990 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 03:17:30.819587 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808993 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 03:17:30.819587 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808996 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 03:17:30.819587 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.808998 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 03:17:30.819587 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.809002 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 03:17:30.819587 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.809004 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 03:17:30.819587 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.809007 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 03:17:30.819587 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.809009 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 03:17:30.819587 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.809012 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 03:17:30.819587 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.809014 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 03:17:30.819587 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.809017 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 03:17:30.819587 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.809019 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 03:17:30.820043 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.809022 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 03:17:30.820043 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.809427 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 03:17:30.820043 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.815960 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 24 03:17:30.820043 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.815977 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 24 03:17:30.820043 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.816482 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 24 03:17:30.820043 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.816491 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 24 03:17:30.820043 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816557 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 03:17:30.820043 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816562 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 03:17:30.820043 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816566 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 03:17:30.820043 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816569 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 03:17:30.820043 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816572 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 03:17:30.820043 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816574 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 03:17:30.820043 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816577 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 03:17:30.820043 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816579 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 03:17:30.820399 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816583 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 03:17:30.820399 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816587 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 03:17:30.820399 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816590 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 03:17:30.820399 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816593 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 03:17:30.820399 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816596 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 03:17:30.820399 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816599 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 03:17:30.820399 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816602 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 03:17:30.820399 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816605 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 03:17:30.820399 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816607 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 03:17:30.820399 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816610 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 03:17:30.820399 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816613 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 03:17:30.820399 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816616 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 03:17:30.820399 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816619 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 03:17:30.820399 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816621 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 03:17:30.820399 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816624 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 03:17:30.820399 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816626 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 03:17:30.820399 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816629 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 03:17:30.820399 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816631 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 03:17:30.820399 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816634 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 03:17:30.820887 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816636 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 03:17:30.820887 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816639 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 03:17:30.820887 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816643 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 03:17:30.820887 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816645 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 03:17:30.820887 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816648 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 03:17:30.820887 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816651 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 03:17:30.820887 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816653 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 03:17:30.820887 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816662 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 03:17:30.820887 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816665 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 03:17:30.820887 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816667 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 03:17:30.820887 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816670 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 03:17:30.820887 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816672 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 03:17:30.820887 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816674 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 03:17:30.820887 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816677 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 03:17:30.820887 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816679 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 03:17:30.820887 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816681 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 03:17:30.820887 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816685 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 03:17:30.820887 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816689 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 03:17:30.820887 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816691 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 03:17:30.821355 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816694 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 03:17:30.821355 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816696 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 03:17:30.821355 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816699 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 03:17:30.821355 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816701 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 03:17:30.821355 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816703 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 03:17:30.821355 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816706 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 24 03:17:30.821355 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816709 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 03:17:30.821355 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816711 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 03:17:30.821355 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816713 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 03:17:30.821355 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816716 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 03:17:30.821355 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816718 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 03:17:30.821355 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816721 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 03:17:30.821355 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816723 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 03:17:30.821355 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816726 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 03:17:30.821355 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816729 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 03:17:30.821355 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816731 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 03:17:30.821355 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816741 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 03:17:30.821355 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816744 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 03:17:30.821355 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816746 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 03:17:30.821355 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816748 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 03:17:30.821889 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816751 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 03:17:30.821889 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816754 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 03:17:30.821889 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816756 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 03:17:30.821889 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816759 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 03:17:30.821889 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816761 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 03:17:30.821889 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816763 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 03:17:30.821889 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816766 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 03:17:30.821889 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816768 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 03:17:30.821889 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816771 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 03:17:30.821889 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816774 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 03:17:30.821889 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816776 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 03:17:30.821889 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816780 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 03:17:30.821889 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816783 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 03:17:30.821889 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816786 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 03:17:30.821889 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816789 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 03:17:30.821889 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816792 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 03:17:30.821889 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816795 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 03:17:30.821889 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816797 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 03:17:30.821889 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816800 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 03:17:30.821889 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816802 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 03:17:30.822361 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816805 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 03:17:30.822361 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816807 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 03:17:30.822361 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816809 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 03:17:30.822361 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816812 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 03:17:30.822361 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816814 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 03:17:30.822361 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816817 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 03:17:30.822361 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816819 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 03:17:30.822361 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816822 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 03:17:30.822361 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816824 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 03:17:30.822361 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816832 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 03:17:30.822361 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816835 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 03:17:30.822361 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816837 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 03:17:30.822361 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816840 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 03:17:30.822361 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.816845 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 03:17:30.822361 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816977 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 03:17:30.822752 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816982 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 03:17:30.822752 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816985 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 03:17:30.822752 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816988 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 03:17:30.822752 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816990 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 03:17:30.822752 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816993 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 03:17:30.822752 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816995 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 03:17:30.822752 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.816998 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 03:17:30.822752 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817001 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 03:17:30.822752 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817003 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 03:17:30.822752 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817005 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 03:17:30.822752 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817008 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 03:17:30.822752 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817010 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 03:17:30.822752 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817012 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 03:17:30.822752 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817015 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 03:17:30.822752 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817017 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 03:17:30.822752 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817020 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 03:17:30.822752 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817022 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 03:17:30.822752 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817025 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 03:17:30.822752 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817027 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 03:17:30.822752 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817029 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 03:17:30.823231 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817032 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 03:17:30.823231 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817034 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 03:17:30.823231 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817036 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 03:17:30.823231 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817039 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 03:17:30.823231 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817041 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 03:17:30.823231 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817044 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 03:17:30.823231 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817046 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 03:17:30.823231 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817055 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 03:17:30.823231 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817058 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 03:17:30.823231 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817060 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 03:17:30.823231 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817063 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 03:17:30.823231 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817065 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 03:17:30.823231 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817068 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 03:17:30.823231 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817070 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 03:17:30.823231 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817072 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 03:17:30.823231 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817075 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 03:17:30.823231 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817077 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 03:17:30.823231 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817080 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 24 03:17:30.823231 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817082 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 03:17:30.823231 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817085 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 03:17:30.823715 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817087 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 03:17:30.823715 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817089 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 03:17:30.823715 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817092 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 03:17:30.823715 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817094 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 03:17:30.823715 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817096 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 03:17:30.823715 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817099 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 03:17:30.823715 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817101 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 03:17:30.823715 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817104 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 03:17:30.823715 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817106 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 03:17:30.823715 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817109 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 03:17:30.823715 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817111 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 03:17:30.823715 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817113 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 03:17:30.823715 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817116 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 03:17:30.823715 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817118 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 03:17:30.823715 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817120 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 03:17:30.823715 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817123 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 03:17:30.823715 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817125 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 03:17:30.823715 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817129 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 03:17:30.823715 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817132 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 03:17:30.823715 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817135 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 03:17:30.824194 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817142 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 03:17:30.824194 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817145 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 03:17:30.824194 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817147 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 03:17:30.824194 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817150 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 03:17:30.824194 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817152 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 03:17:30.824194 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817154 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 03:17:30.824194 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817157 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 03:17:30.824194 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817159 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 03:17:30.824194 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817161 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 03:17:30.824194 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817164 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 03:17:30.824194 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817166 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 03:17:30.824194 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817169 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 03:17:30.824194 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817171 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 03:17:30.824194 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817173 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 03:17:30.824194 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817177 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 03:17:30.824194 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817180 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 03:17:30.824194 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817182 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 03:17:30.824194 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817185 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 03:17:30.824194 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817187 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 03:17:30.824671 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817190 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 03:17:30.824671 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817193 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 03:17:30.824671 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817195 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 03:17:30.824671 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817198 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 03:17:30.824671 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817200 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 03:17:30.824671 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817202 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 03:17:30.824671 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817205 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 03:17:30.824671 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817207 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 03:17:30.824671 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817210 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 03:17:30.824671 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817212 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 03:17:30.824671 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817214 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 03:17:30.824671 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817217 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 03:17:30.824671 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817219 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 03:17:30.824671 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817222 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 03:17:30.824671 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817229 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 03:17:30.824671 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817233 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 03:17:30.824671 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817236 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 03:17:30.824671 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817239 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 03:17:30.824671 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:30.817242 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 03:17:30.825147 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.817246 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 03:17:30.825147 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.817360 2579 server.go:962] "Client rotation is on, will bootstrap in background" Jul 24 03:17:30.825147 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.819282 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 24 03:17:30.825147 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.820720 2579 server.go:1019] "Starting client certificate rotation" Jul 24 03:17:30.825147 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.820847 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 03:17:30.825147 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.820881 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 03:17:30.825147 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.822929 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 03:17:30.825832 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.825814 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 03:17:30.836811 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.836791 2579 log.go:25] "Validated CRI v1 runtime API" Jul 24 03:17:30.838569 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.838554 2579 log.go:25] "Validated CRI v1 image API" Jul 24 03:17:30.839292 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.839276 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 24 03:17:30.841453 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.841419 2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 97e4986a-6f61-4a10-91cf-8f21827c460d:/dev/nvme0n1p4 ecc576ba-78bd-46d6-a4ae-7b994635a4c0:/dev/nvme0n1p3] Jul 24 03:17:30.841548 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.841451 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 24 03:17:30.845429 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.845413 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 03:17:30.846435 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.846324 2579 manager.go:216] Machine: {Timestamp:2026-07-24 03:17:30.845198351 +0000 UTC m=+0.284617801 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3061251 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2637b292ea32caa625d2ce0b93c761 SystemUUID:ec2637b2-92ea-32ca-a625-d2ce0b93c761 BootID:7040852b-c0a8-4204-a09f-9e2273fe4d93 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:07:8c:96:2f:01 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:07:8c:96:2f:01 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:18:97:8d:3b:e7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 24 03:17:30.846435 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.846431 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 24 03:17:30.846537 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.846527 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 24 03:17:30.849317 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.849285 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 24 03:17:30.849469 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.849320 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-147.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 24 03:17:30.850491 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.850479 2579 topology_manager.go:138] "Creating topology manager with none policy" Jul 24 03:17:30.850491 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.850492 2579 container_manager_linux.go:309] "Creating device plugin manager" Jul 24 03:17:30.850562 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.850500 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 03:17:30.851022 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.851012 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 03:17:30.852058 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.852047 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 24 03:17:30.855792 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.855781 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 24 03:17:30.855944 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.855934 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 24 03:17:30.855992 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.855984 2579 kubelet.go:486] "Attempting to sync node with API server" Jul 24 03:17:30.856025 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.855996 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 24 03:17:30.856025 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.856006 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 24 03:17:30.856025 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.856015 2579 kubelet.go:398] "Adding apiserver pod source" Jul 24 03:17:30.856145 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.856027 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 24 03:17:30.856370 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.856358 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 03:17:30.856411 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.856378 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 03:17:30.858586 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.858570 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 24 03:17:30.860252 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.860235 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 24 03:17:30.860331 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.860274 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 24 03:17:30.861584 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.861569 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 24 03:17:30.861667 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.861589 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 24 03:17:30.861667 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.861598 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 24 03:17:30.861667 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.861607 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 24 03:17:30.861667 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.861619 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 24 03:17:30.861667 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.861628 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 24 03:17:30.861667 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.861640 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 24 03:17:30.861667 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.861649 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 24 03:17:30.861667 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.861658 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 24 03:17:30.861667 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.861666 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 24 03:17:30.861923 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.861692 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 24 03:17:30.861923 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.861701 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 24 03:17:30.863078 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.863065 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 24 03:17:30.863138 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.863080 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 24 03:17:30.865674 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.865544 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6skjc" Jul 24 03:17:30.866365 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.866349 2579 server.go:1268] "Started kubelet" Jul 24 03:17:30.866501 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.866451 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 24 03:17:30.866854 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:30.866828 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 24 03:17:30.868806 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.868771 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 24 03:17:30.868806 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.868799 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-147.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 24 03:17:30.868953 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.868829 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 24 03:17:30.869443 ip-10-0-134-147 systemd[1]: Started Kubernetes Kubelet. Jul 24 03:17:30.869589 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:30.869536 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 24 03:17:30.870840 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.870823 2579 server.go:310] "Adding debug handlers to kubelet server" Jul 24 03:17:30.870987 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.870964 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 24 03:17:30.873089 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.873065 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6skjc" Jul 24 03:17:30.873520 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:30.873497 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 24 03:17:30.877269 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:30.876428 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-147.ec2.internal.18c51b7cf79b1e07 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-147.ec2.internal,UID:ip-10-0-134-147.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-147.ec2.internal,},FirstTimestamp:2026-07-24 03:17:30.866302471 +0000 UTC m=+0.305721923,LastTimestamp:2026-07-24 03:17:30.866302471 +0000 UTC m=+0.305721923,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-147.ec2.internal,}" Jul 24 03:17:30.877635 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.877615 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 24 03:17:30.877635 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.877632 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 24 03:17:30.877735 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:30.877702 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-147.ec2.internal\" not found" Jul 24 03:17:30.878198 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.878182 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 24 03:17:30.878299 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.878274 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 24 03:17:30.878299 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.878292 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 24 03:17:30.878417 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.878395 2579 reconstruct.go:97] "Volume reconstruction finished" Jul 24 03:17:30.878417 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.878403 2579 reconciler.go:29] "Reconciler: start to sync state" Jul 24 03:17:30.878947 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.878931 2579 factory.go:55] Registering systemd factory Jul 24 03:17:30.879032 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.878950 2579 factory.go:223] Registration of the systemd container factory successfully Jul 24 03:17:30.879176 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.879161 2579 factory.go:153] Registering CRI-O factory Jul 24 03:17:30.879176 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.879173 2579 factory.go:223] Registration of the crio container factory successfully Jul 24 03:17:30.879286 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.879212 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 24 03:17:30.879286 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.879232 2579 factory.go:103] Registering Raw factory Jul 24 03:17:30.879286 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.879243 2579 manager.go:1196] Started watching for new ooms in manager Jul 24 03:17:30.879597 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.879587 2579 manager.go:349] Starting recovery of all containers Jul 24 03:17:30.883026 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.883005 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 03:17:30.885581 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:30.885557 2579 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-147.ec2.internal\" not found" node="ip-10-0-134-147.ec2.internal" Jul 24 03:17:30.889705 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.889690 2579 manager.go:354] Recovery completed Jul 24 03:17:30.893972 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.893959 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 03:17:30.896078 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.896062 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-147.ec2.internal" event="NodeHasSufficientMemory" Jul 24 03:17:30.896130 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.896098 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 03:17:30.896130 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.896108 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-147.ec2.internal" event="NodeHasSufficientPID" Jul 24 03:17:30.896732 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.896615 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 24 03:17:30.896732 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.896624 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 24 03:17:30.896732 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.896640 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 24 03:17:30.896732 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.896706 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 03:17:30.897501 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.897487 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-147.ec2.internal" event="NodeHasSufficientMemory" Jul 24 03:17:30.897547 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.897513 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 03:17:30.897547 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.897526 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-147.ec2.internal" event="NodeHasSufficientPID" Jul 24 03:17:30.899075 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.899063 2579 policy_none.go:49] "None policy: Start" Jul 24 03:17:30.899116 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.899080 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 24 03:17:30.899116 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.899091 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 24 03:17:30.901587 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.901576 2579 policy_none.go:47] "Start" Jul 24 03:17:30.945155 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.944974 2579 manager.go:337] "Starting Device Plugin manager" Jul 24 03:17:30.945267 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:30.945192 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 24 03:17:30.945267 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.945206 2579 server.go:87] "Starting device plugin registration server" Jul 24 03:17:30.945482 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.945449 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 24 03:17:30.945556 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.945485 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 24 03:17:30.945602 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.945583 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 24 03:17:30.945681 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.945666 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 24 03:17:30.945681 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:30.945680 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 24 03:17:30.946198 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:30.946177 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 24 03:17:30.946265 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:30.946246 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-147.ec2.internal\" not found" Jul 24 03:17:31.025575 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.025535 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 24 03:17:31.027673 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.026920 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 24 03:17:31.027673 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.026958 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 24 03:17:31.027673 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.026981 2579 kubelet.go:2448] "Starting kubelet main sync loop" Jul 24 03:17:31.027673 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:31.027020 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 24 03:17:31.029246 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.029223 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 03:17:31.046265 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.046248 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 03:17:31.047051 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.047033 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-147.ec2.internal" event="NodeHasSufficientMemory" Jul 24 03:17:31.047139 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.047067 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 03:17:31.047139 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.047080 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-147.ec2.internal" event="NodeHasSufficientPID" Jul 24 03:17:31.047139 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.047102 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-147.ec2.internal" Jul 24 03:17:31.054818 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.054759 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-147.ec2.internal" Jul 24 03:17:31.054818 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:31.054784 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-147.ec2.internal\": node \"ip-10-0-134-147.ec2.internal\" not found" Jul 24 03:17:31.070558 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:31.070535 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-147.ec2.internal\" not found" Jul 24 03:17:31.127908 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.127863 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-147.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-147.ec2.internal"] Jul 24 03:17:31.128017 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.127981 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 03:17:31.129503 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.129487 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-147.ec2.internal" event="NodeHasSufficientMemory" Jul 24 03:17:31.129588 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.129521 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 03:17:31.129588 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.129536 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-147.ec2.internal" event="NodeHasSufficientPID" Jul 24 03:17:31.131928 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.131912 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 03:17:31.132052 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.132036 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-147.ec2.internal" Jul 24 03:17:31.132103 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.132069 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 03:17:31.132726 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.132711 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-147.ec2.internal" event="NodeHasSufficientMemory" Jul 24 03:17:31.132786 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.132739 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 03:17:31.132786 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.132748 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-147.ec2.internal" event="NodeHasSufficientPID" Jul 24 03:17:31.133209 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.133192 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-147.ec2.internal" event="NodeHasSufficientMemory" Jul 24 03:17:31.133255 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.133219 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 03:17:31.133255 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.133229 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-147.ec2.internal" event="NodeHasSufficientPID" Jul 24 03:17:31.134910 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.134895 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-147.ec2.internal" Jul 24 03:17:31.134969 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.134920 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 03:17:31.135634 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.135618 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-147.ec2.internal" event="NodeHasSufficientMemory" Jul 24 03:17:31.135718 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.135649 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 03:17:31.135718 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.135662 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-147.ec2.internal" event="NodeHasSufficientPID" Jul 24 03:17:31.149977 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:31.149957 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-147.ec2.internal\" not found" node="ip-10-0-134-147.ec2.internal" Jul 24 03:17:31.153836 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:31.153823 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-147.ec2.internal\" not found" node="ip-10-0-134-147.ec2.internal" Jul 24 03:17:31.171263 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:31.171245 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-147.ec2.internal\" not found" Jul 24 03:17:31.180407 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.180389 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/660bcf57026921c085cc9591248ec280-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-147.ec2.internal\" (UID: \"660bcf57026921c085cc9591248ec280\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-147.ec2.internal" Jul 24 03:17:31.180474 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.180412 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/660bcf57026921c085cc9591248ec280-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-147.ec2.internal\" (UID: \"660bcf57026921c085cc9591248ec280\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-147.ec2.internal" Jul 24 03:17:31.180474 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.180432 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e7160710802e7696127f93542dcf0579-config\") pod \"kube-apiserver-proxy-ip-10-0-134-147.ec2.internal\" (UID: \"e7160710802e7696127f93542dcf0579\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-147.ec2.internal" Jul 24 03:17:31.272295 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:31.272268 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-147.ec2.internal\" not found" Jul 24 03:17:31.280977 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.280950 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/660bcf57026921c085cc9591248ec280-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-147.ec2.internal\" (UID: \"660bcf57026921c085cc9591248ec280\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-147.ec2.internal" Jul 24 03:17:31.281027 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.280985 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/660bcf57026921c085cc9591248ec280-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-147.ec2.internal\" (UID: \"660bcf57026921c085cc9591248ec280\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-147.ec2.internal" Jul 24 03:17:31.281027 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.281000 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e7160710802e7696127f93542dcf0579-config\") pod \"kube-apiserver-proxy-ip-10-0-134-147.ec2.internal\" (UID: \"e7160710802e7696127f93542dcf0579\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-147.ec2.internal" Jul 24 03:17:31.281120 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.281048 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e7160710802e7696127f93542dcf0579-config\") pod \"kube-apiserver-proxy-ip-10-0-134-147.ec2.internal\" (UID: \"e7160710802e7696127f93542dcf0579\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-147.ec2.internal" Jul 24 03:17:31.281120 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.281066 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/660bcf57026921c085cc9591248ec280-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-147.ec2.internal\" (UID: \"660bcf57026921c085cc9591248ec280\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-147.ec2.internal" Jul 24 03:17:31.281120 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.281090 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/660bcf57026921c085cc9591248ec280-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-147.ec2.internal\" (UID: \"660bcf57026921c085cc9591248ec280\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-147.ec2.internal" Jul 24 03:17:31.372994 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:31.372963 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-147.ec2.internal\" not found" Jul 24 03:17:31.451574 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.451534 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-147.ec2.internal" Jul 24 03:17:31.457232 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.457212 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-147.ec2.internal" Jul 24 03:17:31.474031 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:31.474001 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-147.ec2.internal\" not found" Jul 24 03:17:31.574611 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:31.574576 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-147.ec2.internal\" not found" Jul 24 03:17:31.675134 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:31.675055 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-147.ec2.internal\" not found" Jul 24 03:17:31.775735 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:31.775709 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-147.ec2.internal\" not found" Jul 24 03:17:31.786960 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.786930 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 03:17:31.821035 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.821003 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 24 03:17:31.821439 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.821140 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 03:17:31.821439 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.821182 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 03:17:31.821439 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.821182 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 03:17:31.876383 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:31.876358 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-147.ec2.internal\" not found" Jul 24 03:17:31.878412 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.878385 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 03:12:31 +0000 UTC" deadline="2028-02-25 10:23:38.289065074 +0000 UTC" Jul 24 03:17:31.878412 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:31.878408 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13951h6m6.410660635s" Jul 24 03:17:31.976902 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:31.976870 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-147.ec2.internal\" not found" Jul 24 03:17:32.072855 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:32.072809 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660bcf57026921c085cc9591248ec280.slice/crio-9a9b9c12386d7c741224c37f3ce492376287d5e86895cec739e39e536b67633b WatchSource:0}: Error finding container 9a9b9c12386d7c741224c37f3ce492376287d5e86895cec739e39e536b67633b: Status 404 returned error can't find the container with id 9a9b9c12386d7c741224c37f3ce492376287d5e86895cec739e39e536b67633b Jul 24 03:17:32.073418 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:32.073391 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7160710802e7696127f93542dcf0579.slice/crio-ed54456627fc23c1789dda71fd7c0625779aa9736a4ddf2a418f9449a4f1f377 WatchSource:0}: Error finding container ed54456627fc23c1789dda71fd7c0625779aa9736a4ddf2a418f9449a4f1f377: Status 404 returned error can't find the container with id ed54456627fc23c1789dda71fd7c0625779aa9736a4ddf2a418f9449a4f1f377 Jul 24 03:17:32.076995 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:32.076976 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-147.ec2.internal\" not found" Jul 24 03:17:32.077452 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.077441 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 03:17:32.148881 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.148850 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 03:17:32.177971 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.177941 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-147.ec2.internal" Jul 24 03:17:32.188774 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.188748 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 03:17:32.189776 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.189764 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-147.ec2.internal" Jul 24 03:17:32.198226 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.198163 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 03:17:32.857336 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.857302 2579 apiserver.go:52] "Watching apiserver" Jul 24 03:17:32.865002 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.864968 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 24 03:17:32.865452 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.865411 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-s4zpq","openshift-dns/node-resolver-zgsks","openshift-multus/network-metrics-daemon-8xwcn","openshift-network-diagnostics/network-check-target-l4dwr","openshift-network-operator/iptables-alerter-jt59q","openshift-ovn-kubernetes/ovnkube-node-sfxh7","kube-system/global-pull-secret-syncer-fhm9h","kube-system/konnectivity-agent-9g9rf","kube-system/kube-apiserver-proxy-ip-10-0-134-147.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz","openshift-image-registry/node-ca-8brgv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-147.ec2.internal","openshift-multus/multus-additional-cni-plugins-rw2n2","openshift-multus/multus-s7n7v"] Jul 24 03:17:32.868201 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.868180 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:17:32.868313 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:32.868259 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhm9h" podUID="ad666805-ba79-4b67-8a3b-b9ba32afe705" Jul 24 03:17:32.870123 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.870098 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 03:17:32.870547 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.870523 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rw2n2" Jul 24 03:17:32.872964 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.872941 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 24 03:17:32.873174 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.873156 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 24 03:17:32.873429 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.873408 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 24 03:17:32.873606 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.873591 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 24 03:17:32.873852 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.873829 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xnn7b\"" Jul 24 03:17:32.874045 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.874028 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 24 03:17:32.874428 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.874414 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:17:32.874492 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:32.874470 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xwcn" podUID="8fdc01f1-f7a1-4395-bae7-be0c9d76b76c" Jul 24 03:17:32.875024 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.875006 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:17:32.875110 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:32.875087 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4dwr" podUID="cb391e83-8f0f-424a-9f25-8b4a0ecf38d2" Jul 24 03:17:32.877481 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.877447 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jt59q" Jul 24 03:17:32.878329 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.878307 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 24 03:17:32.879815 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.879792 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 24 03:17:32.879815 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.879980 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 24 03:17:32.879815 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.880112 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2smvh\"" Jul 24 03:17:32.880626 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.880607 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 24 03:17:32.880791 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.880767 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.883756 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.882783 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s7n7v" Jul 24 03:17:32.883756 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.882887 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 24 03:17:32.883756 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.883244 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-96hlp\"" Jul 24 03:17:32.883756 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.883387 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 24 03:17:32.883756 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.883452 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 24 03:17:32.883756 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.883542 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 24 03:17:32.883756 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.883604 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 24 03:17:32.883756 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.883666 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 24 03:17:32.885235 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.885086 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 24 03:17:32.885787 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.885665 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-49r5p\"" Jul 24 03:17:32.885940 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.885885 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:32.887634 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.887582 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b31cd8e-39ec-4f6f-80ee-3168270efc37-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rw2n2\" (UID: \"5b31cd8e-39ec-4f6f-80ee-3168270efc37\") " pod="openshift-multus/multus-additional-cni-plugins-rw2n2" Jul 24 03:17:32.887731 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.887639 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-systemd-units\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.887731 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.887664 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-node-log\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.887731 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.887715 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-log-socket\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.887901 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.887749 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f07cdea1-d991-468f-8194-8e4e644d9e8d-env-overrides\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.887901 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.887793 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad666805-ba79-4b67-8a3b-b9ba32afe705-original-pull-secret\") pod \"global-pull-secret-syncer-fhm9h\" (UID: \"ad666805-ba79-4b67-8a3b-b9ba32afe705\") " pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:17:32.887901 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.887821 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5b31cd8e-39ec-4f6f-80ee-3168270efc37-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rw2n2\" (UID: \"5b31cd8e-39ec-4f6f-80ee-3168270efc37\") " pod="openshift-multus/multus-additional-cni-plugins-rw2n2" Jul 24 03:17:32.887901 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.887846 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-host-slash\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.887901 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.887870 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-etc-openvswitch\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.887901 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.887894 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-run-ovn\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.888252 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.887919 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-host-run-ovn-kubernetes\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.888252 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.887957 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-var-lib-openvswitch\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.888252 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.887981 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-run-openvswitch\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.888252 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.888009 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjjwb\" (UniqueName: \"kubernetes.io/projected/5b31cd8e-39ec-4f6f-80ee-3168270efc37-kube-api-access-cjjwb\") pod \"multus-additional-cni-plugins-rw2n2\" (UID: \"5b31cd8e-39ec-4f6f-80ee-3168270efc37\") " pod="openshift-multus/multus-additional-cni-plugins-rw2n2" Jul 24 03:17:32.888252 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.888026 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs\") pod \"network-metrics-daemon-8xwcn\" (UID: \"8fdc01f1-f7a1-4395-bae7-be0c9d76b76c\") " pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:17:32.888252 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.888042 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-host-cni-bin\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.888252 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.888066 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-host-cni-netd\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.888252 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.888098 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ad666805-ba79-4b67-8a3b-b9ba32afe705-dbus\") pod \"global-pull-secret-syncer-fhm9h\" (UID: \"ad666805-ba79-4b67-8a3b-b9ba32afe705\") " pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:17:32.888252 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.888137 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w9vc\" (UniqueName: \"kubernetes.io/projected/b8407b03-4e29-4bbf-8cda-247a5c95175e-kube-api-access-9w9vc\") pod \"iptables-alerter-jt59q\" (UID: \"b8407b03-4e29-4bbf-8cda-247a5c95175e\") " pod="openshift-network-operator/iptables-alerter-jt59q" Jul 24 03:17:32.888252 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.888162 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-host-run-netns\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.888252 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.888185 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b8407b03-4e29-4bbf-8cda-247a5c95175e-iptables-alerter-script\") pod \"iptables-alerter-jt59q\" (UID: \"b8407b03-4e29-4bbf-8cda-247a5c95175e\") " pod="openshift-network-operator/iptables-alerter-jt59q" Jul 24 03:17:32.888252 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.888212 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8407b03-4e29-4bbf-8cda-247a5c95175e-host-slash\") pod \"iptables-alerter-jt59q\" (UID: \"b8407b03-4e29-4bbf-8cda-247a5c95175e\") " pod="openshift-network-operator/iptables-alerter-jt59q" Jul 24 03:17:32.888252 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.888235 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f07cdea1-d991-468f-8194-8e4e644d9e8d-ovnkube-script-lib\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.889058 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.888270 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5b31cd8e-39ec-4f6f-80ee-3168270efc37-os-release\") pod \"multus-additional-cni-plugins-rw2n2\" (UID: \"5b31cd8e-39ec-4f6f-80ee-3168270efc37\") " pod="openshift-multus/multus-additional-cni-plugins-rw2n2" Jul 24 03:17:32.889058 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.888294 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-host-kubelet\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.889058 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.888314 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.889058 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.888330 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f07cdea1-d991-468f-8194-8e4e644d9e8d-ovn-node-metrics-cert\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.889058 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.888344 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-run-systemd\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.889058 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.888383 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ad666805-ba79-4b67-8a3b-b9ba32afe705-kubelet-config\") pod \"global-pull-secret-syncer-fhm9h\" (UID: \"ad666805-ba79-4b67-8a3b-b9ba32afe705\") " pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:17:32.889058 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.888400 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b31cd8e-39ec-4f6f-80ee-3168270efc37-system-cni-dir\") pod \"multus-additional-cni-plugins-rw2n2\" (UID: \"5b31cd8e-39ec-4f6f-80ee-3168270efc37\") " pod="openshift-multus/multus-additional-cni-plugins-rw2n2" Jul 24 03:17:32.889058 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.888418 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b31cd8e-39ec-4f6f-80ee-3168270efc37-cni-binary-copy\") pod \"multus-additional-cni-plugins-rw2n2\" (UID: \"5b31cd8e-39ec-4f6f-80ee-3168270efc37\") " pod="openshift-multus/multus-additional-cni-plugins-rw2n2" Jul 24 03:17:32.889058 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.888433 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5b31cd8e-39ec-4f6f-80ee-3168270efc37-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rw2n2\" (UID: \"5b31cd8e-39ec-4f6f-80ee-3168270efc37\") " pod="openshift-multus/multus-additional-cni-plugins-rw2n2" Jul 24 03:17:32.889058 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.888492 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cst4z\" (UniqueName: \"kubernetes.io/projected/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-kube-api-access-cst4z\") pod \"network-metrics-daemon-8xwcn\" (UID: \"8fdc01f1-f7a1-4395-bae7-be0c9d76b76c\") " pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:17:32.889058 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.888519 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f07cdea1-d991-468f-8194-8e4e644d9e8d-ovnkube-config\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.889058 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.888551 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b31cd8e-39ec-4f6f-80ee-3168270efc37-cnibin\") pod \"multus-additional-cni-plugins-rw2n2\" (UID: \"5b31cd8e-39ec-4f6f-80ee-3168270efc37\") " pod="openshift-multus/multus-additional-cni-plugins-rw2n2" Jul 24 03:17:32.889058 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.888566 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktg2b\" (UniqueName: \"kubernetes.io/projected/cb391e83-8f0f-424a-9f25-8b4a0ecf38d2-kube-api-access-ktg2b\") pod \"network-check-target-l4dwr\" (UID: \"cb391e83-8f0f-424a-9f25-8b4a0ecf38d2\") " pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:17:32.889058 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.888581 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpvvd\" (UniqueName: \"kubernetes.io/projected/f07cdea1-d991-468f-8194-8e4e644d9e8d-kube-api-access-zpvvd\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.890172 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.890150 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 24 03:17:32.890904 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.890881 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zgsks" Jul 24 03:17:32.891881 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.891855 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz" Jul 24 03:17:32.893266 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.893141 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 24 03:17:32.893266 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.893193 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7jrv2\"" Jul 24 03:17:32.894560 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.894525 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8brgv" Jul 24 03:17:32.895614 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.895554 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 24 03:17:32.895826 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.895804 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 24 03:17:32.895956 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.895938 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 24 03:17:32.896028 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.896014 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 24 03:17:32.896087 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.896029 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6kzdt\"" Jul 24 03:17:32.896087 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.896057 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 24 03:17:32.896190 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.896159 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ctsbq\"" Jul 24 03:17:32.896973 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.896917 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 24 03:17:32.897075 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.897056 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 24 03:17:32.897159 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.897131 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-964lf\"" Jul 24 03:17:32.897908 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.897891 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 24 03:17:32.899411 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.899029 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 03:17:32.900751 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.900727 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9g9rf" Jul 24 03:17:32.902987 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.902966 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-nsnr7\"" Jul 24 03:17:32.903065 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.903018 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 24 03:17:32.903065 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.903034 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 24 03:17:32.924600 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.924415 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-p556r" Jul 24 03:17:32.942785 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.942738 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-p556r" Jul 24 03:17:32.979467 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.979437 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 24 03:17:32.989373 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.989333 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ad666805-ba79-4b67-8a3b-b9ba32afe705-kubelet-config\") pod \"global-pull-secret-syncer-fhm9h\" (UID: \"ad666805-ba79-4b67-8a3b-b9ba32afe705\") " pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:17:32.989608 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.989379 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b31cd8e-39ec-4f6f-80ee-3168270efc37-system-cni-dir\") pod \"multus-additional-cni-plugins-rw2n2\" (UID: \"5b31cd8e-39ec-4f6f-80ee-3168270efc37\") " pod="openshift-multus/multus-additional-cni-plugins-rw2n2" Jul 24 03:17:32.989608 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.989408 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5b31cd8e-39ec-4f6f-80ee-3168270efc37-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rw2n2\" (UID: \"5b31cd8e-39ec-4f6f-80ee-3168270efc37\") " pod="openshift-multus/multus-additional-cni-plugins-rw2n2" Jul 24 03:17:32.989608 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.989435 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f07cdea1-d991-468f-8194-8e4e644d9e8d-ovnkube-config\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.989608 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.989463 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-system-cni-dir\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:32.989608 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.989488 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-multus-socket-dir-parent\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:32.989608 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.989489 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ad666805-ba79-4b67-8a3b-b9ba32afe705-kubelet-config\") pod \"global-pull-secret-syncer-fhm9h\" (UID: \"ad666805-ba79-4b67-8a3b-b9ba32afe705\") " pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:17:32.989608 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.989541 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b31cd8e-39ec-4f6f-80ee-3168270efc37-cnibin\") pod \"multus-additional-cni-plugins-rw2n2\" (UID: \"5b31cd8e-39ec-4f6f-80ee-3168270efc37\") " pod="openshift-multus/multus-additional-cni-plugins-rw2n2" Jul 24 03:17:32.989608 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.989569 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktg2b\" (UniqueName: \"kubernetes.io/projected/cb391e83-8f0f-424a-9f25-8b4a0ecf38d2-kube-api-access-ktg2b\") pod \"network-check-target-l4dwr\" (UID: \"cb391e83-8f0f-424a-9f25-8b4a0ecf38d2\") " pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:17:32.989608 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.989594 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-etc-sysctl-d\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:32.989608 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.989618 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-etc-systemd\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:32.989608 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.989641 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-host-var-lib-cni-multus\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:32.990115 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.989667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b31cd8e-39ec-4f6f-80ee-3168270efc37-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rw2n2\" (UID: \"5b31cd8e-39ec-4f6f-80ee-3168270efc37\") " pod="openshift-multus/multus-additional-cni-plugins-rw2n2" Jul 24 03:17:32.990115 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.989696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-log-socket\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.990115 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.989719 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-run\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:32.990115 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.989743 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-sys\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:32.990115 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.989770 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-etc-tuned\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:32.990115 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.989795 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/41c75ca4-e61b-4414-b47b-50490a4b3c4e-sys-fs\") pod \"aws-ebs-csi-driver-node-mrtvz\" (UID: \"41c75ca4-e61b-4414-b47b-50490a4b3c4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz" Jul 24 03:17:32.990115 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.989840 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-host-run-k8s-cni-cncf-io\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:32.990115 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.989881 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad666805-ba79-4b67-8a3b-b9ba32afe705-original-pull-secret\") pod \"global-pull-secret-syncer-fhm9h\" (UID: \"ad666805-ba79-4b67-8a3b-b9ba32afe705\") " pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:17:32.990115 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.989910 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-host-slash\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.990115 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.989938 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgxkt\" (UniqueName: \"kubernetes.io/projected/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-kube-api-access-sgxkt\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:32.990115 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.989963 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-hostroot\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:32.990115 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.989987 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/daacf9bc-9977-408a-b1b4-cd361f3cf528-host\") pod \"node-ca-8brgv\" (UID: \"daacf9bc-9977-408a-b1b4-cd361f3cf528\") " pod="openshift-image-registry/node-ca-8brgv" Jul 24 03:17:32.990115 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990009 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-host\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:32.990115 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990020 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f07cdea1-d991-468f-8194-8e4e644d9e8d-ovnkube-config\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.990115 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990049 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a5b9b325-d029-42ca-8dfb-c75c2dd0c726-hosts-file\") pod \"node-resolver-zgsks\" (UID: \"a5b9b325-d029-42ca-8dfb-c75c2dd0c726\") " pod="openshift-dns/node-resolver-zgsks" Jul 24 03:17:32.990115 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.989495 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b31cd8e-39ec-4f6f-80ee-3168270efc37-system-cni-dir\") pod \"multus-additional-cni-plugins-rw2n2\" (UID: \"5b31cd8e-39ec-4f6f-80ee-3168270efc37\") " pod="openshift-multus/multus-additional-cni-plugins-rw2n2" Jul 24 03:17:32.990115 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990078 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-run-openvswitch\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.990815 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990115 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjjwb\" (UniqueName: \"kubernetes.io/projected/5b31cd8e-39ec-4f6f-80ee-3168270efc37-kube-api-access-cjjwb\") pod \"multus-additional-cni-plugins-rw2n2\" (UID: \"5b31cd8e-39ec-4f6f-80ee-3168270efc37\") " pod="openshift-multus/multus-additional-cni-plugins-rw2n2" Jul 24 03:17:32.990815 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990118 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5b31cd8e-39ec-4f6f-80ee-3168270efc37-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rw2n2\" (UID: \"5b31cd8e-39ec-4f6f-80ee-3168270efc37\") " pod="openshift-multus/multus-additional-cni-plugins-rw2n2" Jul 24 03:17:32.990815 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990164 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs\") pod \"network-metrics-daemon-8xwcn\" (UID: \"8fdc01f1-f7a1-4395-bae7-be0c9d76b76c\") " pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:17:32.990815 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990129 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b31cd8e-39ec-4f6f-80ee-3168270efc37-cnibin\") pod \"multus-additional-cni-plugins-rw2n2\" (UID: \"5b31cd8e-39ec-4f6f-80ee-3168270efc37\") " pod="openshift-multus/multus-additional-cni-plugins-rw2n2" Jul 24 03:17:32.990815 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990191 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-host-cni-bin\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.990815 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990218 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-host-cni-netd\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.990815 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990237 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-log-socket\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.990815 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990246 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-etc-kubernetes\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:32.990815 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990252 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b31cd8e-39ec-4f6f-80ee-3168270efc37-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rw2n2\" (UID: \"5b31cd8e-39ec-4f6f-80ee-3168270efc37\") " pod="openshift-multus/multus-additional-cni-plugins-rw2n2" Jul 24 03:17:32.990815 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990272 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ad666805-ba79-4b67-8a3b-b9ba32afe705-dbus\") pod \"global-pull-secret-syncer-fhm9h\" (UID: \"ad666805-ba79-4b67-8a3b-b9ba32afe705\") " pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:17:32.990815 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990299 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-etc-modprobe-d\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:32.990815 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990337 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-etc-sysctl-conf\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:32.990815 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:32.990344 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 03:17:32.990815 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990401 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-lib-modules\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:32.990815 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990421 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxtjt\" (UniqueName: \"kubernetes.io/projected/41c75ca4-e61b-4414-b47b-50490a4b3c4e-kube-api-access-kxtjt\") pod \"aws-ebs-csi-driver-node-mrtvz\" (UID: \"41c75ca4-e61b-4414-b47b-50490a4b3c4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz" Jul 24 03:17:32.990815 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:32.990433 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 03:17:32.990815 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990438 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-host-var-lib-kubelet\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:32.990815 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990456 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftjcm\" (UniqueName: \"kubernetes.io/projected/daacf9bc-9977-408a-b1b4-cd361f3cf528-kube-api-access-ftjcm\") pod \"node-ca-8brgv\" (UID: \"daacf9bc-9977-408a-b1b4-cd361f3cf528\") " pod="openshift-image-registry/node-ca-8brgv" Jul 24 03:17:32.991670 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990473 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ce6090ae-6c38-4fb6-b7dd-9d76baa4951b-konnectivity-ca\") pod \"konnectivity-agent-9g9rf\" (UID: \"ce6090ae-6c38-4fb6-b7dd-9d76baa4951b\") " pod="kube-system/konnectivity-agent-9g9rf" Jul 24 03:17:32.991670 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:32.990494 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad666805-ba79-4b67-8a3b-b9ba32afe705-original-pull-secret podName:ad666805-ba79-4b67-8a3b-b9ba32afe705 nodeName:}" failed. No retries permitted until 2026-07-24 03:17:33.490472593 +0000 UTC m=+2.929892034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ad666805-ba79-4b67-8a3b-b9ba32afe705-original-pull-secret") pod "global-pull-secret-syncer-fhm9h" (UID: "ad666805-ba79-4b67-8a3b-b9ba32afe705") : object "kube-system"/"original-pull-secret" not registered Jul 24 03:17:32.991670 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990497 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-run-openvswitch\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.991670 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990401 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-host-slash\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.991670 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990567 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ad666805-ba79-4b67-8a3b-b9ba32afe705-dbus\") pod \"global-pull-secret-syncer-fhm9h\" (UID: \"ad666805-ba79-4b67-8a3b-b9ba32afe705\") " pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:17:32.991670 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990596 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f07cdea1-d991-468f-8194-8e4e644d9e8d-ovnkube-script-lib\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.991670 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990641 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-host-var-lib-cni-bin\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:32.991670 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990644 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-host-cni-bin\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.991670 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990682 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-host-cni-netd\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.991670 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:32.990711 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs podName:8fdc01f1-f7a1-4395-bae7-be0c9d76b76c nodeName:}" failed. No retries permitted until 2026-07-24 03:17:33.490684829 +0000 UTC m=+2.930104270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs") pod "network-metrics-daemon-8xwcn" (UID: "8fdc01f1-f7a1-4395-bae7-be0c9d76b76c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 03:17:32.991670 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.990944 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-host-run-multus-certs\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:32.991670 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991000 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-etc-sysconfig\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:32.991670 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991030 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e16c788-ad59-47f8-9cd4-a2a797bb241d-cni-binary-copy\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:32.991670 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991074 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-multus-conf-dir\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:32.991670 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991099 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-run-systemd\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.991670 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991143 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b31cd8e-39ec-4f6f-80ee-3168270efc37-cni-binary-copy\") pod \"multus-additional-cni-plugins-rw2n2\" (UID: \"5b31cd8e-39ec-4f6f-80ee-3168270efc37\") " pod="openshift-multus/multus-additional-cni-plugins-rw2n2" Jul 24 03:17:32.992675 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991185 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-run-systemd\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.992675 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991231 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cst4z\" (UniqueName: \"kubernetes.io/projected/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-kube-api-access-cst4z\") pod \"network-metrics-daemon-8xwcn\" (UID: \"8fdc01f1-f7a1-4395-bae7-be0c9d76b76c\") " pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:17:32.992675 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991252 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f07cdea1-d991-468f-8194-8e4e644d9e8d-ovnkube-script-lib\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.992675 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991340 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twfm2\" (UniqueName: \"kubernetes.io/projected/a5b9b325-d029-42ca-8dfb-c75c2dd0c726-kube-api-access-twfm2\") pod \"node-resolver-zgsks\" (UID: \"a5b9b325-d029-42ca-8dfb-c75c2dd0c726\") " pod="openshift-dns/node-resolver-zgsks" Jul 24 03:17:32.992675 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991454 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/41c75ca4-e61b-4414-b47b-50490a4b3c4e-device-dir\") pod \"aws-ebs-csi-driver-node-mrtvz\" (UID: \"41c75ca4-e61b-4414-b47b-50490a4b3c4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz" Jul 24 03:17:32.992675 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991506 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-host-run-netns\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:32.992675 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991537 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpvvd\" (UniqueName: \"kubernetes.io/projected/f07cdea1-d991-468f-8194-8e4e644d9e8d-kube-api-access-zpvvd\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.992675 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991563 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-var-lib-kubelet\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:32.992675 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991613 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-systemd-units\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.992675 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991657 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-node-log\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.992675 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991687 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f07cdea1-d991-468f-8194-8e4e644d9e8d-env-overrides\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.992675 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991707 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-systemd-units\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.992675 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991717 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/41c75ca4-e61b-4414-b47b-50490a4b3c4e-etc-selinux\") pod \"aws-ebs-csi-driver-node-mrtvz\" (UID: \"41c75ca4-e61b-4414-b47b-50490a4b3c4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz" Jul 24 03:17:32.992675 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991750 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-cnibin\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:32.992675 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991779 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-node-log\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.992675 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991780 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-os-release\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:32.992675 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991826 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5b9b325-d029-42ca-8dfb-c75c2dd0c726-tmp-dir\") pod \"node-resolver-zgsks\" (UID: \"a5b9b325-d029-42ca-8dfb-c75c2dd0c726\") " pod="openshift-dns/node-resolver-zgsks" Jul 24 03:17:32.993816 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991861 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5b31cd8e-39ec-4f6f-80ee-3168270efc37-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rw2n2\" (UID: \"5b31cd8e-39ec-4f6f-80ee-3168270efc37\") " pod="openshift-multus/multus-additional-cni-plugins-rw2n2" Jul 24 03:17:32.993816 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991887 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-etc-openvswitch\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.993816 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991925 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-etc-openvswitch\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.993816 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991925 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-run-ovn\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.993816 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991968 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-host-run-ovn-kubernetes\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.993816 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991977 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-run-ovn\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.993816 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.991997 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/41c75ca4-e61b-4414-b47b-50490a4b3c4e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mrtvz\" (UID: \"41c75ca4-e61b-4414-b47b-50490a4b3c4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz" Jul 24 03:17:32.993816 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992022 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-host-run-ovn-kubernetes\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.993816 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992024 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-multus-cni-dir\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:32.993816 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992052 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-tmp\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:32.993816 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992082 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-var-lib-openvswitch\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.993816 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992105 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/41c75ca4-e61b-4414-b47b-50490a4b3c4e-registration-dir\") pod \"aws-ebs-csi-driver-node-mrtvz\" (UID: \"41c75ca4-e61b-4414-b47b-50490a4b3c4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz" Jul 24 03:17:32.993816 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992136 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/daacf9bc-9977-408a-b1b4-cd361f3cf528-serviceca\") pod \"node-ca-8brgv\" (UID: \"daacf9bc-9977-408a-b1b4-cd361f3cf528\") " pod="openshift-image-registry/node-ca-8brgv" Jul 24 03:17:32.993816 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992180 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-var-lib-openvswitch\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.993816 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992209 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w9vc\" (UniqueName: \"kubernetes.io/projected/b8407b03-4e29-4bbf-8cda-247a5c95175e-kube-api-access-9w9vc\") pod \"iptables-alerter-jt59q\" (UID: \"b8407b03-4e29-4bbf-8cda-247a5c95175e\") " pod="openshift-network-operator/iptables-alerter-jt59q" Jul 24 03:17:32.993816 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992234 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-host-run-netns\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.993816 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992255 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f07cdea1-d991-468f-8194-8e4e644d9e8d-env-overrides\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.994829 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992260 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b8407b03-4e29-4bbf-8cda-247a5c95175e-iptables-alerter-script\") pod \"iptables-alerter-jt59q\" (UID: \"b8407b03-4e29-4bbf-8cda-247a5c95175e\") " pod="openshift-network-operator/iptables-alerter-jt59q" Jul 24 03:17:32.994829 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8407b03-4e29-4bbf-8cda-247a5c95175e-host-slash\") pod \"iptables-alerter-jt59q\" (UID: \"b8407b03-4e29-4bbf-8cda-247a5c95175e\") " pod="openshift-network-operator/iptables-alerter-jt59q" Jul 24 03:17:32.994829 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992314 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/41c75ca4-e61b-4414-b47b-50490a4b3c4e-socket-dir\") pod \"aws-ebs-csi-driver-node-mrtvz\" (UID: \"41c75ca4-e61b-4414-b47b-50490a4b3c4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz" Jul 24 03:17:32.994829 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992319 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-host-run-netns\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.994829 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992352 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wg76\" (UniqueName: \"kubernetes.io/projected/8e16c788-ad59-47f8-9cd4-a2a797bb241d-kube-api-access-8wg76\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:32.994829 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992370 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b31cd8e-39ec-4f6f-80ee-3168270efc37-cni-binary-copy\") pod \"multus-additional-cni-plugins-rw2n2\" (UID: \"5b31cd8e-39ec-4f6f-80ee-3168270efc37\") " pod="openshift-multus/multus-additional-cni-plugins-rw2n2" Jul 24 03:17:32.994829 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992384 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5b31cd8e-39ec-4f6f-80ee-3168270efc37-os-release\") pod \"multus-additional-cni-plugins-rw2n2\" (UID: \"5b31cd8e-39ec-4f6f-80ee-3168270efc37\") " pod="openshift-multus/multus-additional-cni-plugins-rw2n2" Jul 24 03:17:32.994829 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992409 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-host-kubelet\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.994829 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992422 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5b31cd8e-39ec-4f6f-80ee-3168270efc37-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rw2n2\" (UID: \"5b31cd8e-39ec-4f6f-80ee-3168270efc37\") " pod="openshift-multus/multus-additional-cni-plugins-rw2n2" Jul 24 03:17:32.994829 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992446 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.994829 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992459 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8407b03-4e29-4bbf-8cda-247a5c95175e-host-slash\") pod \"iptables-alerter-jt59q\" (UID: \"b8407b03-4e29-4bbf-8cda-247a5c95175e\") " pod="openshift-network-operator/iptables-alerter-jt59q" Jul 24 03:17:32.994829 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992490 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.994829 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992489 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f07cdea1-d991-468f-8194-8e4e644d9e8d-ovn-node-metrics-cert\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.994829 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992521 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f07cdea1-d991-468f-8194-8e4e644d9e8d-host-kubelet\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.994829 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992548 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e16c788-ad59-47f8-9cd4-a2a797bb241d-multus-daemon-config\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:32.994829 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992495 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5b31cd8e-39ec-4f6f-80ee-3168270efc37-os-release\") pod \"multus-additional-cni-plugins-rw2n2\" (UID: \"5b31cd8e-39ec-4f6f-80ee-3168270efc37\") " pod="openshift-multus/multus-additional-cni-plugins-rw2n2" Jul 24 03:17:32.994829 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992572 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-etc-kubernetes\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:32.995781 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992649 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ce6090ae-6c38-4fb6-b7dd-9d76baa4951b-agent-certs\") pod \"konnectivity-agent-9g9rf\" (UID: \"ce6090ae-6c38-4fb6-b7dd-9d76baa4951b\") " pod="kube-system/konnectivity-agent-9g9rf" Jul 24 03:17:32.995781 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992857 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b8407b03-4e29-4bbf-8cda-247a5c95175e-iptables-alerter-script\") pod \"iptables-alerter-jt59q\" (UID: \"b8407b03-4e29-4bbf-8cda-247a5c95175e\") " pod="openshift-network-operator/iptables-alerter-jt59q" Jul 24 03:17:32.995781 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.992871 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 24 03:17:32.997271 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:32.997247 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f07cdea1-d991-468f-8194-8e4e644d9e8d-ovn-node-metrics-cert\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:32.997574 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:32.997555 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 03:17:32.997670 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:32.997579 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 03:17:32.997670 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:32.997593 2579 projected.go:196] Error preparing data for projected volume kube-api-access-ktg2b for pod openshift-network-diagnostics/network-check-target-l4dwr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 03:17:32.997670 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:32.997657 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb391e83-8f0f-424a-9f25-8b4a0ecf38d2-kube-api-access-ktg2b podName:cb391e83-8f0f-424a-9f25-8b4a0ecf38d2 nodeName:}" failed. No retries permitted until 2026-07-24 03:17:33.49763854 +0000 UTC m=+2.937058011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ktg2b" (UniqueName: "kubernetes.io/projected/cb391e83-8f0f-424a-9f25-8b4a0ecf38d2-kube-api-access-ktg2b") pod "network-check-target-l4dwr" (UID: "cb391e83-8f0f-424a-9f25-8b4a0ecf38d2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 03:17:33.001016 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.000995 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjjwb\" (UniqueName: \"kubernetes.io/projected/5b31cd8e-39ec-4f6f-80ee-3168270efc37-kube-api-access-cjjwb\") pod \"multus-additional-cni-plugins-rw2n2\" (UID: \"5b31cd8e-39ec-4f6f-80ee-3168270efc37\") " pod="openshift-multus/multus-additional-cni-plugins-rw2n2" Jul 24 03:17:33.001952 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.001926 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpvvd\" (UniqueName: \"kubernetes.io/projected/f07cdea1-d991-468f-8194-8e4e644d9e8d-kube-api-access-zpvvd\") pod \"ovnkube-node-sfxh7\" (UID: \"f07cdea1-d991-468f-8194-8e4e644d9e8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:33.002043 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.001989 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cst4z\" (UniqueName: \"kubernetes.io/projected/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-kube-api-access-cst4z\") pod \"network-metrics-daemon-8xwcn\" (UID: \"8fdc01f1-f7a1-4395-bae7-be0c9d76b76c\") " pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:17:33.003548 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.003511 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w9vc\" (UniqueName: \"kubernetes.io/projected/b8407b03-4e29-4bbf-8cda-247a5c95175e-kube-api-access-9w9vc\") pod \"iptables-alerter-jt59q\" (UID: \"b8407b03-4e29-4bbf-8cda-247a5c95175e\") " pod="openshift-network-operator/iptables-alerter-jt59q" Jul 24 03:17:33.031774 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.031723 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-147.ec2.internal" event={"ID":"e7160710802e7696127f93542dcf0579","Type":"ContainerStarted","Data":"ed54456627fc23c1789dda71fd7c0625779aa9736a4ddf2a418f9449a4f1f377"} Jul 24 03:17:33.032980 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.032952 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-147.ec2.internal" event={"ID":"660bcf57026921c085cc9591248ec280","Type":"ContainerStarted","Data":"9a9b9c12386d7c741224c37f3ce492376287d5e86895cec739e39e536b67633b"} Jul 24 03:17:33.093740 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.093696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-multus-conf-dir\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.093912 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.093746 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twfm2\" (UniqueName: \"kubernetes.io/projected/a5b9b325-d029-42ca-8dfb-c75c2dd0c726-kube-api-access-twfm2\") pod \"node-resolver-zgsks\" (UID: \"a5b9b325-d029-42ca-8dfb-c75c2dd0c726\") " pod="openshift-dns/node-resolver-zgsks" Jul 24 03:17:33.093912 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.093804 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/41c75ca4-e61b-4414-b47b-50490a4b3c4e-device-dir\") pod \"aws-ebs-csi-driver-node-mrtvz\" (UID: \"41c75ca4-e61b-4414-b47b-50490a4b3c4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz" Jul 24 03:17:33.093912 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.093807 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-multus-conf-dir\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.093912 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.093828 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-host-run-netns\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.093912 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.093897 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/41c75ca4-e61b-4414-b47b-50490a4b3c4e-device-dir\") pod \"aws-ebs-csi-driver-node-mrtvz\" (UID: \"41c75ca4-e61b-4414-b47b-50490a4b3c4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz" Jul 24 03:17:33.094160 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.093926 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-var-lib-kubelet\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.094160 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.093953 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/41c75ca4-e61b-4414-b47b-50490a4b3c4e-etc-selinux\") pod \"aws-ebs-csi-driver-node-mrtvz\" (UID: \"41c75ca4-e61b-4414-b47b-50490a4b3c4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz" Jul 24 03:17:33.094160 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.093978 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-cnibin\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.094160 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.093987 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-host-run-netns\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.094160 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094004 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-os-release\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.094160 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094004 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-var-lib-kubelet\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.094160 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094036 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5b9b325-d029-42ca-8dfb-c75c2dd0c726-tmp-dir\") pod \"node-resolver-zgsks\" (UID: \"a5b9b325-d029-42ca-8dfb-c75c2dd0c726\") " pod="openshift-dns/node-resolver-zgsks" Jul 24 03:17:33.094160 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094068 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-os-release\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.094160 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094120 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/41c75ca4-e61b-4414-b47b-50490a4b3c4e-etc-selinux\") pod \"aws-ebs-csi-driver-node-mrtvz\" (UID: \"41c75ca4-e61b-4414-b47b-50490a4b3c4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz" Jul 24 03:17:33.094160 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094123 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-cnibin\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.094160 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094139 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/41c75ca4-e61b-4414-b47b-50490a4b3c4e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mrtvz\" (UID: \"41c75ca4-e61b-4414-b47b-50490a4b3c4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz" Jul 24 03:17:33.094707 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-multus-cni-dir\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.094707 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094190 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-tmp\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.094707 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094214 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/41c75ca4-e61b-4414-b47b-50490a4b3c4e-registration-dir\") pod \"aws-ebs-csi-driver-node-mrtvz\" (UID: \"41c75ca4-e61b-4414-b47b-50490a4b3c4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz" Jul 24 03:17:33.094707 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094240 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/daacf9bc-9977-408a-b1b4-cd361f3cf528-serviceca\") pod \"node-ca-8brgv\" (UID: \"daacf9bc-9977-408a-b1b4-cd361f3cf528\") " pod="openshift-image-registry/node-ca-8brgv" Jul 24 03:17:33.094707 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094257 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-multus-cni-dir\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.094707 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094267 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/41c75ca4-e61b-4414-b47b-50490a4b3c4e-socket-dir\") pod \"aws-ebs-csi-driver-node-mrtvz\" (UID: \"41c75ca4-e61b-4414-b47b-50490a4b3c4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz" Jul 24 03:17:33.094707 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094311 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/41c75ca4-e61b-4414-b47b-50490a4b3c4e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mrtvz\" (UID: \"41c75ca4-e61b-4414-b47b-50490a4b3c4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz" Jul 24 03:17:33.094707 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094324 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/41c75ca4-e61b-4414-b47b-50490a4b3c4e-registration-dir\") pod \"aws-ebs-csi-driver-node-mrtvz\" (UID: \"41c75ca4-e61b-4414-b47b-50490a4b3c4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz" Jul 24 03:17:33.094707 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094383 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wg76\" (UniqueName: \"kubernetes.io/projected/8e16c788-ad59-47f8-9cd4-a2a797bb241d-kube-api-access-8wg76\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.094707 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094414 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5b9b325-d029-42ca-8dfb-c75c2dd0c726-tmp-dir\") pod \"node-resolver-zgsks\" (UID: \"a5b9b325-d029-42ca-8dfb-c75c2dd0c726\") " pod="openshift-dns/node-resolver-zgsks" Jul 24 03:17:33.094707 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094436 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e16c788-ad59-47f8-9cd4-a2a797bb241d-multus-daemon-config\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.094707 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094457 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-etc-kubernetes\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.094707 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094476 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ce6090ae-6c38-4fb6-b7dd-9d76baa4951b-agent-certs\") pod \"konnectivity-agent-9g9rf\" (UID: \"ce6090ae-6c38-4fb6-b7dd-9d76baa4951b\") " pod="kube-system/konnectivity-agent-9g9rf" Jul 24 03:17:33.094707 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094529 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-system-cni-dir\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.094707 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094556 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-multus-socket-dir-parent\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.094707 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094583 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-etc-kubernetes\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.094707 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094598 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-etc-sysctl-d\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.094707 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094489 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/41c75ca4-e61b-4414-b47b-50490a4b3c4e-socket-dir\") pod \"aws-ebs-csi-driver-node-mrtvz\" (UID: \"41c75ca4-e61b-4414-b47b-50490a4b3c4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz" Jul 24 03:17:33.095780 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094623 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-etc-systemd\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.095780 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094649 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-host-var-lib-cni-multus\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.095780 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094679 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-run\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.095780 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094687 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-system-cni-dir\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.095780 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094701 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-sys\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.095780 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094724 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-etc-tuned\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.095780 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094747 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-multus-socket-dir-parent\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.095780 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094749 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/41c75ca4-e61b-4414-b47b-50490a4b3c4e-sys-fs\") pod \"aws-ebs-csi-driver-node-mrtvz\" (UID: \"41c75ca4-e61b-4414-b47b-50490a4b3c4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz" Jul 24 03:17:33.095780 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094764 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/daacf9bc-9977-408a-b1b4-cd361f3cf528-serviceca\") pod \"node-ca-8brgv\" (UID: \"daacf9bc-9977-408a-b1b4-cd361f3cf528\") " pod="openshift-image-registry/node-ca-8brgv" Jul 24 03:17:33.095780 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094777 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-host-run-k8s-cni-cncf-io\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.095780 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgxkt\" (UniqueName: \"kubernetes.io/projected/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-kube-api-access-sgxkt\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.095780 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094832 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-host-var-lib-cni-multus\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.095780 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094855 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-hostroot\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.095780 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094867 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-etc-systemd\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.095780 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094883 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-run\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.095780 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094888 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-etc-sysctl-d\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.095780 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094922 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-host-run-k8s-cni-cncf-io\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.095780 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094925 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/daacf9bc-9977-408a-b1b4-cd361f3cf528-host\") pod \"node-ca-8brgv\" (UID: \"daacf9bc-9977-408a-b1b4-cd361f3cf528\") " pod="openshift-image-registry/node-ca-8brgv" Jul 24 03:17:33.096694 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094939 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-sys\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.096694 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.094961 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/41c75ca4-e61b-4414-b47b-50490a4b3c4e-sys-fs\") pod \"aws-ebs-csi-driver-node-mrtvz\" (UID: \"41c75ca4-e61b-4414-b47b-50490a4b3c4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz" Jul 24 03:17:33.096694 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.095006 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/daacf9bc-9977-408a-b1b4-cd361f3cf528-host\") pod \"node-ca-8brgv\" (UID: \"daacf9bc-9977-408a-b1b4-cd361f3cf528\") " pod="openshift-image-registry/node-ca-8brgv" Jul 24 03:17:33.096694 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.095031 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-hostroot\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.096694 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.095044 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-host\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.096694 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.095058 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8e16c788-ad59-47f8-9cd4-a2a797bb241d-multus-daemon-config\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.096694 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.095083 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a5b9b325-d029-42ca-8dfb-c75c2dd0c726-hosts-file\") pod \"node-resolver-zgsks\" (UID: \"a5b9b325-d029-42ca-8dfb-c75c2dd0c726\") " pod="openshift-dns/node-resolver-zgsks" Jul 24 03:17:33.096694 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.095096 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-host\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.096694 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.095159 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a5b9b325-d029-42ca-8dfb-c75c2dd0c726-hosts-file\") pod \"node-resolver-zgsks\" (UID: \"a5b9b325-d029-42ca-8dfb-c75c2dd0c726\") " pod="openshift-dns/node-resolver-zgsks" Jul 24 03:17:33.096694 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.095177 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-etc-kubernetes\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.096694 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.095212 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-etc-modprobe-d\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.096694 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.095232 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-etc-kubernetes\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.096694 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.095234 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-etc-sysctl-conf\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.096694 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.095266 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-lib-modules\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.096694 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.095293 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxtjt\" (UniqueName: \"kubernetes.io/projected/41c75ca4-e61b-4414-b47b-50490a4b3c4e-kube-api-access-kxtjt\") pod \"aws-ebs-csi-driver-node-mrtvz\" (UID: \"41c75ca4-e61b-4414-b47b-50490a4b3c4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz" Jul 24 03:17:33.096694 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.095329 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-host-var-lib-kubelet\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.096694 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.095332 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-etc-modprobe-d\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.096694 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.095367 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-etc-sysctl-conf\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.097396 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.095375 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-host-var-lib-kubelet\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.097396 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.095401 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftjcm\" (UniqueName: \"kubernetes.io/projected/daacf9bc-9977-408a-b1b4-cd361f3cf528-kube-api-access-ftjcm\") pod \"node-ca-8brgv\" (UID: \"daacf9bc-9977-408a-b1b4-cd361f3cf528\") " pod="openshift-image-registry/node-ca-8brgv" Jul 24 03:17:33.097396 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.095429 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ce6090ae-6c38-4fb6-b7dd-9d76baa4951b-konnectivity-ca\") pod \"konnectivity-agent-9g9rf\" (UID: \"ce6090ae-6c38-4fb6-b7dd-9d76baa4951b\") " pod="kube-system/konnectivity-agent-9g9rf" Jul 24 03:17:33.097396 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.095457 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-host-var-lib-cni-bin\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.097396 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.095482 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-host-run-multus-certs\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.097396 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.095507 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-etc-sysconfig\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.097396 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.095531 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e16c788-ad59-47f8-9cd4-a2a797bb241d-cni-binary-copy\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.097396 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.095624 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-lib-modules\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.097396 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.095654 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-host-run-multus-certs\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.097396 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.095674 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e16c788-ad59-47f8-9cd4-a2a797bb241d-host-var-lib-cni-bin\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.097396 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.095707 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-etc-sysconfig\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.097396 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.096051 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e16c788-ad59-47f8-9cd4-a2a797bb241d-cni-binary-copy\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.097396 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.096051 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ce6090ae-6c38-4fb6-b7dd-9d76baa4951b-konnectivity-ca\") pod \"konnectivity-agent-9g9rf\" (UID: \"ce6090ae-6c38-4fb6-b7dd-9d76baa4951b\") " pod="kube-system/konnectivity-agent-9g9rf" Jul 24 03:17:33.097396 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.097166 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-tmp\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.097919 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.097798 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-etc-tuned\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.099113 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.099094 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ce6090ae-6c38-4fb6-b7dd-9d76baa4951b-agent-certs\") pod \"konnectivity-agent-9g9rf\" (UID: \"ce6090ae-6c38-4fb6-b7dd-9d76baa4951b\") " pod="kube-system/konnectivity-agent-9g9rf" Jul 24 03:17:33.102526 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.102504 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wg76\" (UniqueName: \"kubernetes.io/projected/8e16c788-ad59-47f8-9cd4-a2a797bb241d-kube-api-access-8wg76\") pod \"multus-s7n7v\" (UID: \"8e16c788-ad59-47f8-9cd4-a2a797bb241d\") " pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.102927 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.102902 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twfm2\" (UniqueName: \"kubernetes.io/projected/a5b9b325-d029-42ca-8dfb-c75c2dd0c726-kube-api-access-twfm2\") pod \"node-resolver-zgsks\" (UID: \"a5b9b325-d029-42ca-8dfb-c75c2dd0c726\") " pod="openshift-dns/node-resolver-zgsks" Jul 24 03:17:33.104050 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.104031 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgxkt\" (UniqueName: \"kubernetes.io/projected/dd37d331-d2d2-4602-83fe-6d6e0cb582fb-kube-api-access-sgxkt\") pod \"tuned-s4zpq\" (UID: \"dd37d331-d2d2-4602-83fe-6d6e0cb582fb\") " pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.104356 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.104323 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftjcm\" (UniqueName: \"kubernetes.io/projected/daacf9bc-9977-408a-b1b4-cd361f3cf528-kube-api-access-ftjcm\") pod \"node-ca-8brgv\" (UID: \"daacf9bc-9977-408a-b1b4-cd361f3cf528\") " pod="openshift-image-registry/node-ca-8brgv" Jul 24 03:17:33.104730 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.104712 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxtjt\" (UniqueName: \"kubernetes.io/projected/41c75ca4-e61b-4414-b47b-50490a4b3c4e-kube-api-access-kxtjt\") pod \"aws-ebs-csi-driver-node-mrtvz\" (UID: \"41c75ca4-e61b-4414-b47b-50490a4b3c4e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz" Jul 24 03:17:33.163168 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.163087 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 03:17:33.184629 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.184591 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rw2n2" Jul 24 03:17:33.193699 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.193673 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jt59q" Jul 24 03:17:33.206192 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.206162 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:17:33.212369 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.212347 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8brgv" Jul 24 03:17:33.225329 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.225298 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s7n7v" Jul 24 03:17:33.233375 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.233350 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" Jul 24 03:17:33.241249 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.241220 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zgsks" Jul 24 03:17:33.249168 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.249148 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz" Jul 24 03:17:33.255053 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.255027 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9g9rf" Jul 24 03:17:33.498780 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.498694 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktg2b\" (UniqueName: \"kubernetes.io/projected/cb391e83-8f0f-424a-9f25-8b4a0ecf38d2-kube-api-access-ktg2b\") pod \"network-check-target-l4dwr\" (UID: \"cb391e83-8f0f-424a-9f25-8b4a0ecf38d2\") " pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:17:33.498780 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.498747 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad666805-ba79-4b67-8a3b-b9ba32afe705-original-pull-secret\") pod \"global-pull-secret-syncer-fhm9h\" (UID: \"ad666805-ba79-4b67-8a3b-b9ba32afe705\") " pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:17:33.498780 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.498783 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs\") pod \"network-metrics-daemon-8xwcn\" (UID: \"8fdc01f1-f7a1-4395-bae7-be0c9d76b76c\") " pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:17:33.499073 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:33.498837 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 03:17:33.499073 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:33.498857 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 03:17:33.499073 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:33.498870 2579 projected.go:196] Error preparing data for projected volume kube-api-access-ktg2b for pod openshift-network-diagnostics/network-check-target-l4dwr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 03:17:33.499073 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:33.498896 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 03:17:33.499073 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:33.498932 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb391e83-8f0f-424a-9f25-8b4a0ecf38d2-kube-api-access-ktg2b podName:cb391e83-8f0f-424a-9f25-8b4a0ecf38d2 nodeName:}" failed. No retries permitted until 2026-07-24 03:17:34.498913753 +0000 UTC m=+3.938333214 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktg2b" (UniqueName: "kubernetes.io/projected/cb391e83-8f0f-424a-9f25-8b4a0ecf38d2-kube-api-access-ktg2b") pod "network-check-target-l4dwr" (UID: "cb391e83-8f0f-424a-9f25-8b4a0ecf38d2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 03:17:33.499073 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:33.498955 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs podName:8fdc01f1-f7a1-4395-bae7-be0c9d76b76c nodeName:}" failed. No retries permitted until 2026-07-24 03:17:34.498943981 +0000 UTC m=+3.938363431 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs") pod "network-metrics-daemon-8xwcn" (UID: "8fdc01f1-f7a1-4395-bae7-be0c9d76b76c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 03:17:33.499073 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:33.498970 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 03:17:33.499073 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:33.498999 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad666805-ba79-4b67-8a3b-b9ba32afe705-original-pull-secret podName:ad666805-ba79-4b67-8a3b-b9ba32afe705 nodeName:}" failed. No retries permitted until 2026-07-24 03:17:34.498988963 +0000 UTC m=+3.938408403 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ad666805-ba79-4b67-8a3b-b9ba32afe705-original-pull-secret") pod "global-pull-secret-syncer-fhm9h" (UID: "ad666805-ba79-4b67-8a3b-b9ba32afe705") : object "kube-system"/"original-pull-secret" not registered Jul 24 03:17:33.639015 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.638822 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 03:17:33.946732 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.946689 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 03:12:33 +0000 UTC" deadline="2028-01-29 10:37:36.692173343 +0000 UTC" Jul 24 03:17:33.946732 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:33.946729 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13303h20m2.745449819s" Jul 24 03:17:34.027212 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:34.027178 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:17:34.027352 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:34.027287 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4dwr" podUID="cb391e83-8f0f-424a-9f25-8b4a0ecf38d2" Jul 24 03:17:34.071607 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:34.071441 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8407b03_4e29_4bbf_8cda_247a5c95175e.slice/crio-868b257ee3f7e6fb13c98d9c154650cafc74551cd321672e1f8b5cfd830c5f64 WatchSource:0}: Error finding container 868b257ee3f7e6fb13c98d9c154650cafc74551cd321672e1f8b5cfd830c5f64: Status 404 returned error can't find the container with id 868b257ee3f7e6fb13c98d9c154650cafc74551cd321672e1f8b5cfd830c5f64 Jul 24 03:17:34.080798 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:34.080766 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e16c788_ad59_47f8_9cd4_a2a797bb241d.slice/crio-b8cdc1aeaf515a394d5b498e764b0b67af5310747955426235d0dce7dccc9ef3 WatchSource:0}: Error finding container b8cdc1aeaf515a394d5b498e764b0b67af5310747955426235d0dce7dccc9ef3: Status 404 returned error can't find the container with id b8cdc1aeaf515a394d5b498e764b0b67af5310747955426235d0dce7dccc9ef3 Jul 24 03:17:34.082846 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:34.082164 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b31cd8e_39ec_4f6f_80ee_3168270efc37.slice/crio-1f1a3bd99d568b66828cf9c6ff9e7d4da31622262ae15b1e9c190116a41f3687 WatchSource:0}: Error finding container 1f1a3bd99d568b66828cf9c6ff9e7d4da31622262ae15b1e9c190116a41f3687: Status 404 returned error can't find the container with id 1f1a3bd99d568b66828cf9c6ff9e7d4da31622262ae15b1e9c190116a41f3687 Jul 24 03:17:34.083066 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:34.083040 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b9b325_d029_42ca_8dfb_c75c2dd0c726.slice/crio-2e9e385a262728fdee8cdc1a6ec13c9f4c025ce6092bca0be71171c0f50d1520 WatchSource:0}: Error finding container 2e9e385a262728fdee8cdc1a6ec13c9f4c025ce6092bca0be71171c0f50d1520: Status 404 returned error can't find the container with id 2e9e385a262728fdee8cdc1a6ec13c9f4c025ce6092bca0be71171c0f50d1520 Jul 24 03:17:34.085692 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:17:34.085363 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce6090ae_6c38_4fb6_b7dd_9d76baa4951b.slice/crio-15f614c6cf1ce266eb8cac0bdc38177d9e954ff24b74bfff304a34b64259bc9e WatchSource:0}: Error finding container 15f614c6cf1ce266eb8cac0bdc38177d9e954ff24b74bfff304a34b64259bc9e: Status 404 returned error can't find the container with id 15f614c6cf1ce266eb8cac0bdc38177d9e954ff24b74bfff304a34b64259bc9e Jul 24 03:17:34.508079 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:34.507784 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs\") pod \"network-metrics-daemon-8xwcn\" (UID: \"8fdc01f1-f7a1-4395-bae7-be0c9d76b76c\") " pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:17:34.508079 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:34.508055 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktg2b\" (UniqueName: \"kubernetes.io/projected/cb391e83-8f0f-424a-9f25-8b4a0ecf38d2-kube-api-access-ktg2b\") pod \"network-check-target-l4dwr\" (UID: \"cb391e83-8f0f-424a-9f25-8b4a0ecf38d2\") " pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:17:34.508079 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:34.507974 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 03:17:34.508364 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:34.508089 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad666805-ba79-4b67-8a3b-b9ba32afe705-original-pull-secret\") pod \"global-pull-secret-syncer-fhm9h\" (UID: \"ad666805-ba79-4b67-8a3b-b9ba32afe705\") " pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:17:34.508364 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:34.508142 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs podName:8fdc01f1-f7a1-4395-bae7-be0c9d76b76c nodeName:}" failed. No retries permitted until 2026-07-24 03:17:36.508124189 +0000 UTC m=+5.947543633 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs") pod "network-metrics-daemon-8xwcn" (UID: "8fdc01f1-f7a1-4395-bae7-be0c9d76b76c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 03:17:34.508364 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:34.508206 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 03:17:34.508364 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:34.508253 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 03:17:34.508364 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:34.508266 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad666805-ba79-4b67-8a3b-b9ba32afe705-original-pull-secret podName:ad666805-ba79-4b67-8a3b-b9ba32afe705 nodeName:}" failed. No retries permitted until 2026-07-24 03:17:36.508248894 +0000 UTC m=+5.947668352 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ad666805-ba79-4b67-8a3b-b9ba32afe705-original-pull-secret") pod "global-pull-secret-syncer-fhm9h" (UID: "ad666805-ba79-4b67-8a3b-b9ba32afe705") : object "kube-system"/"original-pull-secret" not registered Jul 24 03:17:34.508364 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:34.508280 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 03:17:34.508364 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:34.508297 2579 projected.go:196] Error preparing data for projected volume kube-api-access-ktg2b for pod openshift-network-diagnostics/network-check-target-l4dwr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 03:17:34.508364 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:34.508345 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb391e83-8f0f-424a-9f25-8b4a0ecf38d2-kube-api-access-ktg2b podName:cb391e83-8f0f-424a-9f25-8b4a0ecf38d2 nodeName:}" failed. No retries permitted until 2026-07-24 03:17:36.508334152 +0000 UTC m=+5.947753597 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktg2b" (UniqueName: "kubernetes.io/projected/cb391e83-8f0f-424a-9f25-8b4a0ecf38d2-kube-api-access-ktg2b") pod "network-check-target-l4dwr" (UID: "cb391e83-8f0f-424a-9f25-8b4a0ecf38d2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 03:17:34.948715 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:34.948652 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 03:12:33 +0000 UTC" deadline="2028-03-11 14:06:06.995042941 +0000 UTC" Jul 24 03:17:34.948715 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:34.948712 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14314h48m32.046335654s" Jul 24 03:17:35.031571 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:35.030636 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:17:35.031571 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:35.030770 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhm9h" podUID="ad666805-ba79-4b67-8a3b-b9ba32afe705" Jul 24 03:17:35.031571 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:35.031175 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:17:35.031571 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:35.031289 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xwcn" podUID="8fdc01f1-f7a1-4395-bae7-be0c9d76b76c" Jul 24 03:17:35.055241 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:35.055207 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-147.ec2.internal" event={"ID":"e7160710802e7696127f93542dcf0579","Type":"ContainerStarted","Data":"8371191e288cd0ac19e5dc7f025298cd97cf89d9dd815a7083fc8e2914aace03"} Jul 24 03:17:35.073740 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:35.073675 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-147.ec2.internal" podStartSLOduration=3.07365514 podStartE2EDuration="3.07365514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 03:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:17:35.072751215 +0000 UTC m=+4.512170681" watchObservedRunningTime="2026-07-24 03:17:35.07365514 +0000 UTC m=+4.513074606" Jul 24 03:17:35.079563 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:35.079433 2579 generic.go:356] "Generic (PLEG): container finished" podID="660bcf57026921c085cc9591248ec280" containerID="777a9f2a7e2a9d54468420cc2bb7cac3aa88340b1f8bfb9506e9524c581d0da8" exitCode=0 Jul 24 03:17:35.079563 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:35.079557 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-147.ec2.internal" event={"ID":"660bcf57026921c085cc9591248ec280","Type":"ContainerDied","Data":"777a9f2a7e2a9d54468420cc2bb7cac3aa88340b1f8bfb9506e9524c581d0da8"} Jul 24 03:17:35.100565 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:35.100486 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" event={"ID":"dd37d331-d2d2-4602-83fe-6d6e0cb582fb","Type":"ContainerStarted","Data":"01578df920a1e58c8a59c722c15127c8a22072dae4faa3d765928ad713688910"} Jul 24 03:17:35.132737 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:35.132701 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8brgv" event={"ID":"daacf9bc-9977-408a-b1b4-cd361f3cf528","Type":"ContainerStarted","Data":"47f47fda6a64c498fc52205540c8958449d2a9a485f5d7ad01a6a8d13d8c15d2"} Jul 24 03:17:35.139295 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:35.139172 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz" event={"ID":"41c75ca4-e61b-4414-b47b-50490a4b3c4e","Type":"ContainerStarted","Data":"7160623acce5612dd98883aff7aef661c12e68e719dba9bc100d6c88a0ee8961"} Jul 24 03:17:35.154661 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:35.154623 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9g9rf" event={"ID":"ce6090ae-6c38-4fb6-b7dd-9d76baa4951b","Type":"ContainerStarted","Data":"15f614c6cf1ce266eb8cac0bdc38177d9e954ff24b74bfff304a34b64259bc9e"} Jul 24 03:17:35.165064 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:35.165023 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rw2n2" event={"ID":"5b31cd8e-39ec-4f6f-80ee-3168270efc37","Type":"ContainerStarted","Data":"1f1a3bd99d568b66828cf9c6ff9e7d4da31622262ae15b1e9c190116a41f3687"} Jul 24 03:17:35.172104 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:35.170910 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" event={"ID":"f07cdea1-d991-468f-8194-8e4e644d9e8d","Type":"ContainerStarted","Data":"7ac95e797eaff8684f3c763df7f796ff9c341366ba1ae6ba4bbe1d1a770de002"} Jul 24 03:17:35.187793 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:35.187752 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zgsks" event={"ID":"a5b9b325-d029-42ca-8dfb-c75c2dd0c726","Type":"ContainerStarted","Data":"2e9e385a262728fdee8cdc1a6ec13c9f4c025ce6092bca0be71171c0f50d1520"} Jul 24 03:17:35.196836 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:35.196798 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s7n7v" event={"ID":"8e16c788-ad59-47f8-9cd4-a2a797bb241d","Type":"ContainerStarted","Data":"b8cdc1aeaf515a394d5b498e764b0b67af5310747955426235d0dce7dccc9ef3"} Jul 24 03:17:35.200156 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:35.200071 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jt59q" event={"ID":"b8407b03-4e29-4bbf-8cda-247a5c95175e","Type":"ContainerStarted","Data":"868b257ee3f7e6fb13c98d9c154650cafc74551cd321672e1f8b5cfd830c5f64"} Jul 24 03:17:36.027765 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:36.027729 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:17:36.028216 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:36.027864 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4dwr" podUID="cb391e83-8f0f-424a-9f25-8b4a0ecf38d2" Jul 24 03:17:36.216302 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:36.216229 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-147.ec2.internal" event={"ID":"660bcf57026921c085cc9591248ec280","Type":"ContainerStarted","Data":"f3204d1b066b5239e8dd048f36cf4fba2cf3b5115f3b9245dcbd8e03a8f586ed"} Jul 24 03:17:36.233310 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:36.232102 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-147.ec2.internal" podStartSLOduration=4.232080168 podStartE2EDuration="4.232080168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 03:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:17:36.23176422 +0000 UTC m=+5.671183687" watchObservedRunningTime="2026-07-24 03:17:36.232080168 +0000 UTC m=+5.671499620" Jul 24 03:17:36.527751 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:36.526857 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktg2b\" (UniqueName: \"kubernetes.io/projected/cb391e83-8f0f-424a-9f25-8b4a0ecf38d2-kube-api-access-ktg2b\") pod \"network-check-target-l4dwr\" (UID: \"cb391e83-8f0f-424a-9f25-8b4a0ecf38d2\") " pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:17:36.527751 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:36.526912 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad666805-ba79-4b67-8a3b-b9ba32afe705-original-pull-secret\") pod \"global-pull-secret-syncer-fhm9h\" (UID: \"ad666805-ba79-4b67-8a3b-b9ba32afe705\") " pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:17:36.527751 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:36.526948 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs\") pod \"network-metrics-daemon-8xwcn\" (UID: \"8fdc01f1-f7a1-4395-bae7-be0c9d76b76c\") " pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:17:36.527751 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:36.527106 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 03:17:36.527751 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:36.527170 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs podName:8fdc01f1-f7a1-4395-bae7-be0c9d76b76c nodeName:}" failed. No retries permitted until 2026-07-24 03:17:40.527150422 +0000 UTC m=+9.966569870 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs") pod "network-metrics-daemon-8xwcn" (UID: "8fdc01f1-f7a1-4395-bae7-be0c9d76b76c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 03:17:36.527751 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:36.527628 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 03:17:36.527751 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:36.527654 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 03:17:36.527751 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:36.527668 2579 projected.go:196] Error preparing data for projected volume kube-api-access-ktg2b for pod openshift-network-diagnostics/network-check-target-l4dwr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 03:17:36.527751 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:36.527671 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 03:17:36.527751 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:36.527717 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb391e83-8f0f-424a-9f25-8b4a0ecf38d2-kube-api-access-ktg2b podName:cb391e83-8f0f-424a-9f25-8b4a0ecf38d2 nodeName:}" failed. No retries permitted until 2026-07-24 03:17:40.527699436 +0000 UTC m=+9.967118880 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktg2b" (UniqueName: "kubernetes.io/projected/cb391e83-8f0f-424a-9f25-8b4a0ecf38d2-kube-api-access-ktg2b") pod "network-check-target-l4dwr" (UID: "cb391e83-8f0f-424a-9f25-8b4a0ecf38d2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 03:17:36.527751 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:36.527745 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad666805-ba79-4b67-8a3b-b9ba32afe705-original-pull-secret podName:ad666805-ba79-4b67-8a3b-b9ba32afe705 nodeName:}" failed. No retries permitted until 2026-07-24 03:17:40.527724971 +0000 UTC m=+9.967144426 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ad666805-ba79-4b67-8a3b-b9ba32afe705-original-pull-secret") pod "global-pull-secret-syncer-fhm9h" (UID: "ad666805-ba79-4b67-8a3b-b9ba32afe705") : object "kube-system"/"original-pull-secret" not registered Jul 24 03:17:37.029360 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:37.028630 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:17:37.029360 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:37.028773 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhm9h" podUID="ad666805-ba79-4b67-8a3b-b9ba32afe705" Jul 24 03:17:37.029360 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:37.029195 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:17:37.029360 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:37.029308 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xwcn" podUID="8fdc01f1-f7a1-4395-bae7-be0c9d76b76c" Jul 24 03:17:38.028280 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:38.027746 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:17:38.028280 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:38.027904 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4dwr" podUID="cb391e83-8f0f-424a-9f25-8b4a0ecf38d2" Jul 24 03:17:39.028309 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:39.027768 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:17:39.028309 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:39.027920 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhm9h" podUID="ad666805-ba79-4b67-8a3b-b9ba32afe705" Jul 24 03:17:39.029149 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:39.028552 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:17:39.029149 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:39.028680 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xwcn" podUID="8fdc01f1-f7a1-4395-bae7-be0c9d76b76c" Jul 24 03:17:40.027284 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:40.027237 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:17:40.027497 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:40.027401 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4dwr" podUID="cb391e83-8f0f-424a-9f25-8b4a0ecf38d2" Jul 24 03:17:40.564834 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:40.564790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad666805-ba79-4b67-8a3b-b9ba32afe705-original-pull-secret\") pod \"global-pull-secret-syncer-fhm9h\" (UID: \"ad666805-ba79-4b67-8a3b-b9ba32afe705\") " pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:17:40.565276 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:40.564856 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs\") pod \"network-metrics-daemon-8xwcn\" (UID: \"8fdc01f1-f7a1-4395-bae7-be0c9d76b76c\") " pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:17:40.565276 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:40.564944 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktg2b\" (UniqueName: \"kubernetes.io/projected/cb391e83-8f0f-424a-9f25-8b4a0ecf38d2-kube-api-access-ktg2b\") pod \"network-check-target-l4dwr\" (UID: \"cb391e83-8f0f-424a-9f25-8b4a0ecf38d2\") " pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:17:40.565276 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:40.565106 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 03:17:40.565276 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:40.565133 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 03:17:40.565276 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:40.565148 2579 projected.go:196] Error preparing data for projected volume kube-api-access-ktg2b for pod openshift-network-diagnostics/network-check-target-l4dwr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 03:17:40.565276 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:40.565216 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb391e83-8f0f-424a-9f25-8b4a0ecf38d2-kube-api-access-ktg2b podName:cb391e83-8f0f-424a-9f25-8b4a0ecf38d2 nodeName:}" failed. No retries permitted until 2026-07-24 03:17:48.565192866 +0000 UTC m=+18.004612325 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktg2b" (UniqueName: "kubernetes.io/projected/cb391e83-8f0f-424a-9f25-8b4a0ecf38d2-kube-api-access-ktg2b") pod "network-check-target-l4dwr" (UID: "cb391e83-8f0f-424a-9f25-8b4a0ecf38d2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 03:17:40.565670 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:40.565651 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 03:17:40.565757 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:40.565700 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad666805-ba79-4b67-8a3b-b9ba32afe705-original-pull-secret podName:ad666805-ba79-4b67-8a3b-b9ba32afe705 nodeName:}" failed. No retries permitted until 2026-07-24 03:17:48.565687572 +0000 UTC m=+18.005107009 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ad666805-ba79-4b67-8a3b-b9ba32afe705-original-pull-secret") pod "global-pull-secret-syncer-fhm9h" (UID: "ad666805-ba79-4b67-8a3b-b9ba32afe705") : object "kube-system"/"original-pull-secret" not registered Jul 24 03:17:40.565757 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:40.565738 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 03:17:40.565757 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:40.565758 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs podName:8fdc01f1-f7a1-4395-bae7-be0c9d76b76c nodeName:}" failed. No retries permitted until 2026-07-24 03:17:48.565750974 +0000 UTC m=+18.005170412 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs") pod "network-metrics-daemon-8xwcn" (UID: "8fdc01f1-f7a1-4395-bae7-be0c9d76b76c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 03:17:41.028607 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:41.028144 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:17:41.028607 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:41.028258 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhm9h" podUID="ad666805-ba79-4b67-8a3b-b9ba32afe705" Jul 24 03:17:41.028607 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:41.028318 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:17:41.028607 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:41.028427 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xwcn" podUID="8fdc01f1-f7a1-4395-bae7-be0c9d76b76c" Jul 24 03:17:42.027617 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:42.027585 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:17:42.028053 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:42.027715 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4dwr" podUID="cb391e83-8f0f-424a-9f25-8b4a0ecf38d2" Jul 24 03:17:43.028287 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:43.028239 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:17:43.028287 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:43.028290 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:17:43.028816 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:43.028411 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xwcn" podUID="8fdc01f1-f7a1-4395-bae7-be0c9d76b76c" Jul 24 03:17:43.028816 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:43.028588 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhm9h" podUID="ad666805-ba79-4b67-8a3b-b9ba32afe705" Jul 24 03:17:44.027880 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:44.027845 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:17:44.028049 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:44.027956 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4dwr" podUID="cb391e83-8f0f-424a-9f25-8b4a0ecf38d2" Jul 24 03:17:45.028615 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:45.028581 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:17:45.029060 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:45.028581 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:17:45.029060 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:45.028716 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xwcn" podUID="8fdc01f1-f7a1-4395-bae7-be0c9d76b76c" Jul 24 03:17:45.029060 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:45.028745 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhm9h" podUID="ad666805-ba79-4b67-8a3b-b9ba32afe705" Jul 24 03:17:46.027949 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:46.027786 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:17:46.027949 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:46.027909 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4dwr" podUID="cb391e83-8f0f-424a-9f25-8b4a0ecf38d2" Jul 24 03:17:47.027433 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:47.027397 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:17:47.027884 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:47.027557 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhm9h" podUID="ad666805-ba79-4b67-8a3b-b9ba32afe705" Jul 24 03:17:47.027884 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:47.027614 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:17:47.027884 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:47.027733 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xwcn" podUID="8fdc01f1-f7a1-4395-bae7-be0c9d76b76c" Jul 24 03:17:48.027371 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:48.027335 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:17:48.027627 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:48.027480 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4dwr" podUID="cb391e83-8f0f-424a-9f25-8b4a0ecf38d2" Jul 24 03:17:48.628439 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:48.628395 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktg2b\" (UniqueName: \"kubernetes.io/projected/cb391e83-8f0f-424a-9f25-8b4a0ecf38d2-kube-api-access-ktg2b\") pod \"network-check-target-l4dwr\" (UID: \"cb391e83-8f0f-424a-9f25-8b4a0ecf38d2\") " pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:17:48.628743 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:48.628485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad666805-ba79-4b67-8a3b-b9ba32afe705-original-pull-secret\") pod \"global-pull-secret-syncer-fhm9h\" (UID: \"ad666805-ba79-4b67-8a3b-b9ba32afe705\") " pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:17:48.628743 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:48.628511 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs\") pod \"network-metrics-daemon-8xwcn\" (UID: \"8fdc01f1-f7a1-4395-bae7-be0c9d76b76c\") " pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:17:48.628743 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:48.628600 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 03:17:48.628743 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:48.628629 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 03:17:48.628743 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:48.628629 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 03:17:48.628743 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:48.628636 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 03:17:48.628743 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:48.628644 2579 projected.go:196] Error preparing data for projected volume kube-api-access-ktg2b for pod openshift-network-diagnostics/network-check-target-l4dwr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 03:17:48.628743 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:48.628689 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad666805-ba79-4b67-8a3b-b9ba32afe705-original-pull-secret podName:ad666805-ba79-4b67-8a3b-b9ba32afe705 nodeName:}" failed. No retries permitted until 2026-07-24 03:18:04.628672937 +0000 UTC m=+34.068092378 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ad666805-ba79-4b67-8a3b-b9ba32afe705-original-pull-secret") pod "global-pull-secret-syncer-fhm9h" (UID: "ad666805-ba79-4b67-8a3b-b9ba32afe705") : object "kube-system"/"original-pull-secret" not registered Jul 24 03:17:48.628743 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:48.628703 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs podName:8fdc01f1-f7a1-4395-bae7-be0c9d76b76c nodeName:}" failed. No retries permitted until 2026-07-24 03:18:04.628696323 +0000 UTC m=+34.068115761 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs") pod "network-metrics-daemon-8xwcn" (UID: "8fdc01f1-f7a1-4395-bae7-be0c9d76b76c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 03:17:48.628743 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:48.628713 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb391e83-8f0f-424a-9f25-8b4a0ecf38d2-kube-api-access-ktg2b podName:cb391e83-8f0f-424a-9f25-8b4a0ecf38d2 nodeName:}" failed. No retries permitted until 2026-07-24 03:18:04.628707511 +0000 UTC m=+34.068126949 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktg2b" (UniqueName: "kubernetes.io/projected/cb391e83-8f0f-424a-9f25-8b4a0ecf38d2-kube-api-access-ktg2b") pod "network-check-target-l4dwr" (UID: "cb391e83-8f0f-424a-9f25-8b4a0ecf38d2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 03:17:49.028535 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:49.028216 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:17:49.028535 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:49.028362 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhm9h" podUID="ad666805-ba79-4b67-8a3b-b9ba32afe705" Jul 24 03:17:49.028535 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:49.028408 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:17:49.029029 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:49.028552 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xwcn" podUID="8fdc01f1-f7a1-4395-bae7-be0c9d76b76c" Jul 24 03:17:50.027960 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:50.027925 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:17:50.028189 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:50.028041 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4dwr" podUID="cb391e83-8f0f-424a-9f25-8b4a0ecf38d2" Jul 24 03:17:51.028603 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:51.028570 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:17:51.029083 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:51.028673 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhm9h" podUID="ad666805-ba79-4b67-8a3b-b9ba32afe705" Jul 24 03:17:51.029083 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:51.028733 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:17:51.029083 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:51.028807 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xwcn" podUID="8fdc01f1-f7a1-4395-bae7-be0c9d76b76c" Jul 24 03:17:52.027398 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:52.027364 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:17:52.027598 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:52.027492 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4dwr" podUID="cb391e83-8f0f-424a-9f25-8b4a0ecf38d2" Jul 24 03:17:53.027706 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:53.027676 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:17:53.028176 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:53.027803 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhm9h" podUID="ad666805-ba79-4b67-8a3b-b9ba32afe705" Jul 24 03:17:53.028176 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:53.027676 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:17:53.028298 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:53.028281 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xwcn" podUID="8fdc01f1-f7a1-4395-bae7-be0c9d76b76c" Jul 24 03:17:54.028368 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:54.028027 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:17:54.029111 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:54.028478 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4dwr" podUID="cb391e83-8f0f-424a-9f25-8b4a0ecf38d2" Jul 24 03:17:54.263249 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:54.263211 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz" event={"ID":"41c75ca4-e61b-4414-b47b-50490a4b3c4e","Type":"ContainerStarted","Data":"45e11f54f82fcca4546caa314e055ab541e8ff1a231486cb3c78cc7f923b6f1d"} Jul 24 03:17:54.264664 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:54.264640 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9g9rf" event={"ID":"ce6090ae-6c38-4fb6-b7dd-9d76baa4951b","Type":"ContainerStarted","Data":"5351b88a8e73dfeb640b6c79d10b1ce46936b569575a1f99fdafb1c2a18fa69f"} Jul 24 03:17:54.266250 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:54.266227 2579 generic.go:356] "Generic (PLEG): container finished" podID="5b31cd8e-39ec-4f6f-80ee-3168270efc37" containerID="37b2d017149b249e3349bb06aa256a1fce7b739cab0078c68e663073579c68a6" exitCode=0 Jul 24 03:17:54.266351 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:54.266285 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rw2n2" event={"ID":"5b31cd8e-39ec-4f6f-80ee-3168270efc37","Type":"ContainerDied","Data":"37b2d017149b249e3349bb06aa256a1fce7b739cab0078c68e663073579c68a6"} Jul 24 03:17:54.269838 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:54.269811 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" event={"ID":"f07cdea1-d991-468f-8194-8e4e644d9e8d","Type":"ContainerStarted","Data":"f753648b0f8abf8bb5610ed02c7ecabfade2473805f1bcb792b250a200d6182b"} Jul 24 03:17:54.269926 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:54.269890 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" event={"ID":"f07cdea1-d991-468f-8194-8e4e644d9e8d","Type":"ContainerStarted","Data":"62a69c59976192b86fe65d4375c473c182ecfef8aef07f2d8d3bdd77ba6e1b18"} Jul 24 03:17:54.269926 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:54.269924 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" event={"ID":"f07cdea1-d991-468f-8194-8e4e644d9e8d","Type":"ContainerStarted","Data":"400a5e078a6910f39b0dc58de113f3108c1e87d1346b0aaccc97de3b4fa328ce"} Jul 24 03:17:54.270007 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:54.269951 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" event={"ID":"f07cdea1-d991-468f-8194-8e4e644d9e8d","Type":"ContainerStarted","Data":"fa2de9a8ebca4f2dff357b57c2d40b2d1f54cac90f0217bd70b1180324f595cb"} Jul 24 03:17:54.270007 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:54.269979 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" event={"ID":"f07cdea1-d991-468f-8194-8e4e644d9e8d","Type":"ContainerStarted","Data":"59a814c4e64631df2eeff44fbf0e8126b2db805d367aaab3ed9709d25b92be37"} Jul 24 03:17:54.270007 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:54.270004 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" event={"ID":"f07cdea1-d991-468f-8194-8e4e644d9e8d","Type":"ContainerStarted","Data":"91dec02b17850436fa8ac8694430f68ac3e6a1411f1695ec97d2f2780ef4857e"} Jul 24 03:17:54.271580 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:54.271561 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zgsks" event={"ID":"a5b9b325-d029-42ca-8dfb-c75c2dd0c726","Type":"ContainerStarted","Data":"be9623bbc67e778827a71e1288f2f9cf52ed3e405ec2527ebc8678d662afa3a3"} Jul 24 03:17:54.273282 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:54.273257 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s7n7v" event={"ID":"8e16c788-ad59-47f8-9cd4-a2a797bb241d","Type":"ContainerStarted","Data":"44c7251225fed0eb4d50a63c5c6eb175744328defb303d7d6fd48a7ebe4d1d10"} Jul 24 03:17:54.274969 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:54.274948 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" event={"ID":"dd37d331-d2d2-4602-83fe-6d6e0cb582fb","Type":"ContainerStarted","Data":"8acc4c090d02106a1fbc96dd1e99b9483c0861868bad42b2a1c5ef3004ba8b24"} Jul 24 03:17:54.276510 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:54.276483 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8brgv" event={"ID":"daacf9bc-9977-408a-b1b4-cd361f3cf528","Type":"ContainerStarted","Data":"d254b51d86262a8db4dec9b76a577363e3aa5d135894b1839f16853e5ca152a6"} Jul 24 03:17:54.280676 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:54.280627 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9g9rf" podStartSLOduration=8.030722141 podStartE2EDuration="23.280609057s" totalImagesPullingTime="15.249886916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:17:54.280087104 +0000 UTC m=+23.719506568" watchObservedRunningTime="2026-07-24 03:17:54.280609057 +0000 UTC m=+23.720028519" Jul 24 03:17:54.314273 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:54.314229 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-s4zpq" podStartSLOduration=4.14062769 podStartE2EDuration="23.314214506s" totalImagesPullingTime="19.173586816s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 03:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:17:54.314136445 +0000 UTC m=+23.753555905" watchObservedRunningTime="2026-07-24 03:17:54.314214506 +0000 UTC m=+23.753633966" Jul 24 03:17:54.330397 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:54.330352 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zgsks" podStartSLOduration=4.173121586 podStartE2EDuration="23.330338347s" totalImagesPullingTime="19.157216761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 03:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:17:54.330087257 +0000 UTC m=+23.769506716" watchObservedRunningTime="2026-07-24 03:17:54.330338347 +0000 UTC m=+23.769757806" Jul 24 03:17:54.347412 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:54.347373 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-s7n7v" podStartSLOduration=4.128116253 podStartE2EDuration="23.347361193s" totalImagesPullingTime="19.21924494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 03:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:17:54.346968011 +0000 UTC m=+23.786387470" watchObservedRunningTime="2026-07-24 03:17:54.347361193 +0000 UTC m=+23.786780653" Jul 24 03:17:54.360711 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:54.360669 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8brgv" podStartSLOduration=4.209883515 podStartE2EDuration="23.360657209s" totalImagesPullingTime="19.150773694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 03:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:17:54.360267951 +0000 UTC m=+23.799687411" watchObservedRunningTime="2026-07-24 03:17:54.360657209 +0000 UTC m=+23.800076668" Jul 24 03:17:54.491574 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:54.491546 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:17:54.546566 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:54.546542 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:17:54.646054 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:54.646034 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 24 03:17:54.961148 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:54.961018 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T03:17:54.646048431Z","UUID":"f2fd4dcc-5f91-4465-bb20-cc38bc67238a","Handler":null,"Name":"","Endpoint":""} Jul 24 03:17:54.962580 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:54.962554 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 24 03:17:54.962705 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:54.962595 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 24 03:17:55.028113 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:55.028083 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:17:55.028272 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:55.028226 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhm9h" podUID="ad666805-ba79-4b67-8a3b-b9ba32afe705" Jul 24 03:17:55.028341 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:55.028286 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:17:55.028434 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:55.028416 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xwcn" podUID="8fdc01f1-f7a1-4395-bae7-be0c9d76b76c" Jul 24 03:17:55.035536 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:55.035493 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:17:55.112806 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:55.112759 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:17:55.280873 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:55.280792 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jt59q" event={"ID":"b8407b03-4e29-4bbf-8cda-247a5c95175e","Type":"ContainerStarted","Data":"2b5525788a18b2ffd02f7646fd1c196ca812865c9f35c58fb7d74bed6748c0a4"} Jul 24 03:17:55.286926 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:55.286888 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz" event={"ID":"41c75ca4-e61b-4414-b47b-50490a4b3c4e","Type":"ContainerStarted","Data":"f4d3642afdcaffe837ccfaf7abd9aa85b192a1ef502bcc1c6559aaf4a2069ba6"} Jul 24 03:17:55.297708 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:55.297658 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jt59q" podStartSLOduration=5.132538992 podStartE2EDuration="24.297643186s" totalImagesPullingTime="19.165104194s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 03:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:17:55.297295961 +0000 UTC m=+24.736715422" watchObservedRunningTime="2026-07-24 03:17:55.297643186 +0000 UTC m=+24.737062648" Jul 24 03:17:55.413842 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:55.413808 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:17:55.514509 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:55.514472 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:17:55.819173 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:55.819136 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:17:55.881188 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:55.881163 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:17:56.027626 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:56.027588 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:17:56.027780 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:56.027712 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4dwr" podUID="cb391e83-8f0f-424a-9f25-8b4a0ecf38d2" Jul 24 03:17:56.291442 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:56.291398 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" event={"ID":"f07cdea1-d991-468f-8194-8e4e644d9e8d","Type":"ContainerStarted","Data":"fd74ba25d360a0e361be0e9d5152b7ea4fcf0b099193ee29fc44e8f5b1d1775b"} Jul 24 03:17:56.294192 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:56.294161 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz" event={"ID":"41c75ca4-e61b-4414-b47b-50490a4b3c4e","Type":"ContainerStarted","Data":"e7c60801745f44f7f3bf78ac234f3ff5a52855869d14d8d46801403838c5bc5e"} Jul 24 03:17:56.311822 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:56.311776 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mrtvz" podStartSLOduration=3.825509295 podStartE2EDuration="25.311759902s" totalImagesPullingTime="21.486250607s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 03:17:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:17:56.31077102 +0000 UTC m=+25.750190480" watchObservedRunningTime="2026-07-24 03:17:56.311759902 +0000 UTC m=+25.751179363" Jul 24 03:17:57.028166 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:57.028131 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:17:57.028166 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:57.028132 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:17:57.028364 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:57.028269 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhm9h" podUID="ad666805-ba79-4b67-8a3b-b9ba32afe705" Jul 24 03:17:57.028405 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:57.028366 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xwcn" podUID="8fdc01f1-f7a1-4395-bae7-be0c9d76b76c" Jul 24 03:17:57.718990 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:57.718745 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9g9rf" Jul 24 03:17:57.719608 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:57.719586 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9g9rf" Jul 24 03:17:58.027843 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:58.027774 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:17:58.027987 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:58.027885 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4dwr" podUID="cb391e83-8f0f-424a-9f25-8b4a0ecf38d2" Jul 24 03:17:58.299396 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:58.299321 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9g9rf" Jul 24 03:17:58.299870 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:58.299854 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9g9rf" Jul 24 03:17:58.370788 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:58.370751 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:17:59.027403 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:59.027374 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:17:59.027935 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:59.027374 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:17:59.027935 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:59.027528 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xwcn" podUID="8fdc01f1-f7a1-4395-bae7-be0c9d76b76c" Jul 24 03:17:59.027935 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:17:59.027563 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhm9h" podUID="ad666805-ba79-4b67-8a3b-b9ba32afe705" Jul 24 03:17:59.305227 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:59.305189 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" event={"ID":"f07cdea1-d991-468f-8194-8e4e644d9e8d","Type":"ContainerStarted","Data":"72c256daa55b947f58affc959823c4f86c44d7424cbecfd0207e98c26f2a1ac1"} Jul 24 03:17:59.348966 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:17:59.348914 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" podStartSLOduration=9.110681621 podStartE2EDuration="28.348898171s" totalImagesPullingTime="19.23821655s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 03:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:17:59.34877253 +0000 UTC m=+28.788192000" watchObservedRunningTime="2026-07-24 03:17:59.348898171 +0000 UTC m=+28.788317632" Jul 24 03:18:00.027794 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:00.027609 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:18:00.028409 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:00.027860 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4dwr" podUID="cb391e83-8f0f-424a-9f25-8b4a0ecf38d2" Jul 24 03:18:00.309308 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:00.309223 2579 generic.go:356] "Generic (PLEG): container finished" podID="5b31cd8e-39ec-4f6f-80ee-3168270efc37" containerID="b17a33106cc4d0e012c49dbf4dd3b18da3106188bb6c79a8d289b9ddc3de7620" exitCode=0 Jul 24 03:18:00.309533 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:00.309314 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rw2n2" event={"ID":"5b31cd8e-39ec-4f6f-80ee-3168270efc37","Type":"ContainerDied","Data":"b17a33106cc4d0e012c49dbf4dd3b18da3106188bb6c79a8d289b9ddc3de7620"} Jul 24 03:18:00.310751 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:00.310269 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:18:00.310751 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:00.310323 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:18:00.310751 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:00.310352 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:18:00.325764 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:00.325740 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:18:00.325906 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:00.325869 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:18:00.494443 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:00.494411 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:18:00.561387 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:00.561306 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:18:00.616783 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:00.616752 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:18:01.028368 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:01.028338 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:18:01.028806 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:01.028417 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:18:01.028806 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:01.028424 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhm9h" podUID="ad666805-ba79-4b67-8a3b-b9ba32afe705" Jul 24 03:18:01.028806 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:01.028587 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xwcn" podUID="8fdc01f1-f7a1-4395-bae7-be0c9d76b76c" Jul 24 03:18:01.315690 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:01.315664 2579 generic.go:356] "Generic (PLEG): container finished" podID="5b31cd8e-39ec-4f6f-80ee-3168270efc37" containerID="8d08086dd111d114a1e96f5d52254e0309b7e79ff0cc514e732414e541eaa413" exitCode=0 Jul 24 03:18:01.315823 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:01.315749 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rw2n2" event={"ID":"5b31cd8e-39ec-4f6f-80ee-3168270efc37","Type":"ContainerDied","Data":"8d08086dd111d114a1e96f5d52254e0309b7e79ff0cc514e732414e541eaa413"} Jul 24 03:18:01.422049 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:01.422009 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l4dwr"] Jul 24 03:18:01.422221 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:01.422123 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:18:01.422221 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:01.422207 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4dwr" podUID="cb391e83-8f0f-424a-9f25-8b4a0ecf38d2" Jul 24 03:18:01.425009 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:01.424986 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fhm9h"] Jul 24 03:18:01.425160 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:01.425076 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:18:01.425221 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:01.425176 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhm9h" podUID="ad666805-ba79-4b67-8a3b-b9ba32afe705" Jul 24 03:18:01.425544 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:01.425524 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8xwcn"] Jul 24 03:18:01.425613 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:01.425603 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:18:01.425697 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:01.425682 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xwcn" podUID="8fdc01f1-f7a1-4395-bae7-be0c9d76b76c" Jul 24 03:18:03.027919 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:03.027695 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:18:03.028286 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:03.027703 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:18:03.028286 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:03.028003 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhm9h" podUID="ad666805-ba79-4b67-8a3b-b9ba32afe705" Jul 24 03:18:03.028286 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:03.027745 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:18:03.028286 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:03.028079 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xwcn" podUID="8fdc01f1-f7a1-4395-bae7-be0c9d76b76c" Jul 24 03:18:03.028286 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:03.028132 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4dwr" podUID="cb391e83-8f0f-424a-9f25-8b4a0ecf38d2" Jul 24 03:18:03.326344 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:03.326311 2579 generic.go:356] "Generic (PLEG): container finished" podID="5b31cd8e-39ec-4f6f-80ee-3168270efc37" containerID="d205ffaf5bb3e7f5760087ac34fc4441c9a01a2a34cca78d7f5194c7c79bae9b" exitCode=0 Jul 24 03:18:03.326549 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:03.326355 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rw2n2" event={"ID":"5b31cd8e-39ec-4f6f-80ee-3168270efc37","Type":"ContainerDied","Data":"d205ffaf5bb3e7f5760087ac34fc4441c9a01a2a34cca78d7f5194c7c79bae9b"} Jul 24 03:18:04.652369 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:04.652320 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktg2b\" (UniqueName: \"kubernetes.io/projected/cb391e83-8f0f-424a-9f25-8b4a0ecf38d2-kube-api-access-ktg2b\") pod \"network-check-target-l4dwr\" (UID: \"cb391e83-8f0f-424a-9f25-8b4a0ecf38d2\") " pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:18:04.652971 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:04.652383 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad666805-ba79-4b67-8a3b-b9ba32afe705-original-pull-secret\") pod \"global-pull-secret-syncer-fhm9h\" (UID: \"ad666805-ba79-4b67-8a3b-b9ba32afe705\") " pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:18:04.652971 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:04.652419 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs\") pod \"network-metrics-daemon-8xwcn\" (UID: \"8fdc01f1-f7a1-4395-bae7-be0c9d76b76c\") " pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:18:04.652971 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:04.652504 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 03:18:04.652971 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:04.652532 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 03:18:04.652971 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:04.652532 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 03:18:04.652971 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:04.652547 2579 projected.go:196] Error preparing data for projected volume kube-api-access-ktg2b for pod openshift-network-diagnostics/network-check-target-l4dwr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 03:18:04.652971 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:04.652566 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 03:18:04.652971 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:04.652623 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs podName:8fdc01f1-f7a1-4395-bae7-be0c9d76b76c nodeName:}" failed. No retries permitted until 2026-07-24 03:18:36.652606397 +0000 UTC m=+66.092025850 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs") pod "network-metrics-daemon-8xwcn" (UID: "8fdc01f1-f7a1-4395-bae7-be0c9d76b76c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 03:18:04.652971 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:04.652638 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb391e83-8f0f-424a-9f25-8b4a0ecf38d2-kube-api-access-ktg2b podName:cb391e83-8f0f-424a-9f25-8b4a0ecf38d2 nodeName:}" failed. No retries permitted until 2026-07-24 03:18:36.652630994 +0000 UTC m=+66.092050432 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktg2b" (UniqueName: "kubernetes.io/projected/cb391e83-8f0f-424a-9f25-8b4a0ecf38d2-kube-api-access-ktg2b") pod "network-check-target-l4dwr" (UID: "cb391e83-8f0f-424a-9f25-8b4a0ecf38d2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 03:18:04.652971 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:04.652647 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad666805-ba79-4b67-8a3b-b9ba32afe705-original-pull-secret podName:ad666805-ba79-4b67-8a3b-b9ba32afe705 nodeName:}" failed. No retries permitted until 2026-07-24 03:18:36.652642624 +0000 UTC m=+66.092062063 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ad666805-ba79-4b67-8a3b-b9ba32afe705-original-pull-secret") pod "global-pull-secret-syncer-fhm9h" (UID: "ad666805-ba79-4b67-8a3b-b9ba32afe705") : object "kube-system"/"original-pull-secret" not registered Jul 24 03:18:05.027729 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:05.027633 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:18:05.027729 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:05.027653 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:18:05.027956 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:05.027634 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:18:05.027956 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:05.027773 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fhm9h" podUID="ad666805-ba79-4b67-8a3b-b9ba32afe705" Jul 24 03:18:05.027956 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:05.027875 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xwcn" podUID="8fdc01f1-f7a1-4395-bae7-be0c9d76b76c" Jul 24 03:18:05.027956 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:05.027945 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l4dwr" podUID="cb391e83-8f0f-424a-9f25-8b4a0ecf38d2" Jul 24 03:18:05.118921 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:05.118874 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:18:05.219111 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:05.219079 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:18:05.678472 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:05.678425 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:18:05.732645 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:05.732610 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:18:05.925287 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:05.925251 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-147.ec2.internal" event="NodeReady" Jul 24 03:18:05.925472 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:05.925392 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 24 03:18:05.964455 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:05.964388 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4zq5x"] Jul 24 03:18:05.968032 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:05.968013 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-75ljs"] Jul 24 03:18:05.968139 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:05.968122 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4zq5x" Jul 24 03:18:05.970442 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:05.970419 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fzmf8\"" Jul 24 03:18:05.970562 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:05.970475 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 24 03:18:05.970562 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:05.970419 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 24 03:18:05.971450 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:05.971427 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-75ljs" Jul 24 03:18:05.973612 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:05.973595 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 24 03:18:05.973720 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:05.973597 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 24 03:18:05.973720 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:05.973693 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-k789z\"" Jul 24 03:18:05.973720 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:05.973647 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 24 03:18:05.981148 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:05.981078 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4zq5x"] Jul 24 03:18:05.982034 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:05.982009 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-75ljs"] Jul 24 03:18:06.064569 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:06.064536 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl99r\" (UniqueName: \"kubernetes.io/projected/e63f0954-7da9-4fdc-a43d-d1cc3323aaa1-kube-api-access-sl99r\") pod \"ingress-canary-75ljs\" (UID: \"e63f0954-7da9-4fdc-a43d-d1cc3323aaa1\") " pod="openshift-ingress-canary/ingress-canary-75ljs" Jul 24 03:18:06.064754 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:06.064584 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-tmp-dir\") pod \"dns-default-4zq5x\" (UID: \"1a36f9a1-1cc8-4858-be3f-ec11e34eebab\") " pod="openshift-dns/dns-default-4zq5x" Jul 24 03:18:06.064754 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:06.064613 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls\") pod \"dns-default-4zq5x\" (UID: \"1a36f9a1-1cc8-4858-be3f-ec11e34eebab\") " pod="openshift-dns/dns-default-4zq5x" Jul 24 03:18:06.064754 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:06.064669 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdhcx\" (UniqueName: \"kubernetes.io/projected/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-kube-api-access-cdhcx\") pod \"dns-default-4zq5x\" (UID: \"1a36f9a1-1cc8-4858-be3f-ec11e34eebab\") " pod="openshift-dns/dns-default-4zq5x" Jul 24 03:18:06.064754 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:06.064722 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-config-volume\") pod \"dns-default-4zq5x\" (UID: \"1a36f9a1-1cc8-4858-be3f-ec11e34eebab\") " pod="openshift-dns/dns-default-4zq5x" Jul 24 03:18:06.064754 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:06.064747 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e63f0954-7da9-4fdc-a43d-d1cc3323aaa1-cert\") pod \"ingress-canary-75ljs\" (UID: \"e63f0954-7da9-4fdc-a43d-d1cc3323aaa1\") " pod="openshift-ingress-canary/ingress-canary-75ljs" Jul 24 03:18:06.089241 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:06.089208 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:18:06.140320 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:06.140292 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:18:06.165326 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:06.165288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl99r\" (UniqueName: \"kubernetes.io/projected/e63f0954-7da9-4fdc-a43d-d1cc3323aaa1-kube-api-access-sl99r\") pod \"ingress-canary-75ljs\" (UID: \"e63f0954-7da9-4fdc-a43d-d1cc3323aaa1\") " pod="openshift-ingress-canary/ingress-canary-75ljs" Jul 24 03:18:06.165326 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:06.165331 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-tmp-dir\") pod \"dns-default-4zq5x\" (UID: \"1a36f9a1-1cc8-4858-be3f-ec11e34eebab\") " pod="openshift-dns/dns-default-4zq5x" Jul 24 03:18:06.165580 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:06.165368 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls\") pod \"dns-default-4zq5x\" (UID: \"1a36f9a1-1cc8-4858-be3f-ec11e34eebab\") " pod="openshift-dns/dns-default-4zq5x" Jul 24 03:18:06.165580 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:06.165406 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdhcx\" (UniqueName: \"kubernetes.io/projected/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-kube-api-access-cdhcx\") pod \"dns-default-4zq5x\" (UID: \"1a36f9a1-1cc8-4858-be3f-ec11e34eebab\") " pod="openshift-dns/dns-default-4zq5x" Jul 24 03:18:06.165580 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:06.165479 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-config-volume\") pod \"dns-default-4zq5x\" (UID: \"1a36f9a1-1cc8-4858-be3f-ec11e34eebab\") " pod="openshift-dns/dns-default-4zq5x" Jul 24 03:18:06.165580 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:06.165504 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e63f0954-7da9-4fdc-a43d-d1cc3323aaa1-cert\") pod \"ingress-canary-75ljs\" (UID: \"e63f0954-7da9-4fdc-a43d-d1cc3323aaa1\") " pod="openshift-ingress-canary/ingress-canary-75ljs" Jul 24 03:18:06.165580 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:06.165544 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 03:18:06.165818 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:06.165607 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls podName:1a36f9a1-1cc8-4858-be3f-ec11e34eebab nodeName:}" failed. No retries permitted until 2026-07-24 03:18:06.665591032 +0000 UTC m=+36.105010470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls") pod "dns-default-4zq5x" (UID: "1a36f9a1-1cc8-4858-be3f-ec11e34eebab") : secret "dns-default-metrics-tls" not found Jul 24 03:18:06.165818 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:06.165756 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-tmp-dir\") pod \"dns-default-4zq5x\" (UID: \"1a36f9a1-1cc8-4858-be3f-ec11e34eebab\") " pod="openshift-dns/dns-default-4zq5x" Jul 24 03:18:06.166243 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:06.166214 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-config-volume\") pod \"dns-default-4zq5x\" (UID: \"1a36f9a1-1cc8-4858-be3f-ec11e34eebab\") " pod="openshift-dns/dns-default-4zq5x" Jul 24 03:18:06.170201 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:06.170175 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e63f0954-7da9-4fdc-a43d-d1cc3323aaa1-cert\") pod \"ingress-canary-75ljs\" (UID: \"e63f0954-7da9-4fdc-a43d-d1cc3323aaa1\") " pod="openshift-ingress-canary/ingress-canary-75ljs" Jul 24 03:18:06.176384 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:06.176362 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdhcx\" (UniqueName: \"kubernetes.io/projected/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-kube-api-access-cdhcx\") pod \"dns-default-4zq5x\" (UID: \"1a36f9a1-1cc8-4858-be3f-ec11e34eebab\") " pod="openshift-dns/dns-default-4zq5x" Jul 24 03:18:06.176941 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:06.176916 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl99r\" (UniqueName: \"kubernetes.io/projected/e63f0954-7da9-4fdc-a43d-d1cc3323aaa1-kube-api-access-sl99r\") pod \"ingress-canary-75ljs\" (UID: \"e63f0954-7da9-4fdc-a43d-d1cc3323aaa1\") " pod="openshift-ingress-canary/ingress-canary-75ljs" Jul 24 03:18:06.284852 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:06.284777 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-75ljs" Jul 24 03:18:06.669610 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:06.669565 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls\") pod \"dns-default-4zq5x\" (UID: \"1a36f9a1-1cc8-4858-be3f-ec11e34eebab\") " pod="openshift-dns/dns-default-4zq5x" Jul 24 03:18:06.669798 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:06.669716 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 03:18:06.669858 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:06.669801 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls podName:1a36f9a1-1cc8-4858-be3f-ec11e34eebab nodeName:}" failed. No retries permitted until 2026-07-24 03:18:07.669776588 +0000 UTC m=+37.109196044 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls") pod "dns-default-4zq5x" (UID: "1a36f9a1-1cc8-4858-be3f-ec11e34eebab") : secret "dns-default-metrics-tls" not found Jul 24 03:18:07.027814 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:07.027734 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:18:07.027814 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:07.027779 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:18:07.028578 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:07.027741 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:18:07.030700 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:07.030676 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 03:18:07.030700 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:07.030698 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n6kfx\"" Jul 24 03:18:07.031751 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:07.031729 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 03:18:07.031751 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:07.031748 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-frl8x\"" Jul 24 03:18:07.031925 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:07.031764 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 03:18:07.031925 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:07.031904 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 03:18:07.617248 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:07.617069 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-75ljs"] Jul 24 03:18:07.623560 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:18:07.623527 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode63f0954_7da9_4fdc_a43d_d1cc3323aaa1.slice/crio-ca8a0e9d17e28af71c362f7a75a795e19885ee98c4f0b4d334d7ef4c9bc35021 WatchSource:0}: Error finding container ca8a0e9d17e28af71c362f7a75a795e19885ee98c4f0b4d334d7ef4c9bc35021: Status 404 returned error can't find the container with id ca8a0e9d17e28af71c362f7a75a795e19885ee98c4f0b4d334d7ef4c9bc35021 Jul 24 03:18:07.676814 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:07.676777 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls\") pod \"dns-default-4zq5x\" (UID: \"1a36f9a1-1cc8-4858-be3f-ec11e34eebab\") " pod="openshift-dns/dns-default-4zq5x" Jul 24 03:18:07.676981 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:07.676943 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 03:18:07.677035 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:07.677013 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls podName:1a36f9a1-1cc8-4858-be3f-ec11e34eebab nodeName:}" failed. No retries permitted until 2026-07-24 03:18:09.676994337 +0000 UTC m=+39.116413775 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls") pod "dns-default-4zq5x" (UID: "1a36f9a1-1cc8-4858-be3f-ec11e34eebab") : secret "dns-default-metrics-tls" not found Jul 24 03:18:08.343363 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:08.343239 2579 generic.go:356] "Generic (PLEG): container finished" podID="5b31cd8e-39ec-4f6f-80ee-3168270efc37" containerID="120c6712288a7d0a4daad73da83c6b37b5c3992c08bae503a63fe2a4db583fc7" exitCode=0 Jul 24 03:18:08.343363 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:08.343320 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rw2n2" event={"ID":"5b31cd8e-39ec-4f6f-80ee-3168270efc37","Type":"ContainerDied","Data":"120c6712288a7d0a4daad73da83c6b37b5c3992c08bae503a63fe2a4db583fc7"} Jul 24 03:18:08.344943 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:08.344873 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-75ljs" event={"ID":"e63f0954-7da9-4fdc-a43d-d1cc3323aaa1","Type":"ContainerStarted","Data":"ca8a0e9d17e28af71c362f7a75a795e19885ee98c4f0b4d334d7ef4c9bc35021"} Jul 24 03:18:09.350834 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:09.350795 2579 generic.go:356] "Generic (PLEG): container finished" podID="5b31cd8e-39ec-4f6f-80ee-3168270efc37" containerID="434e57b2ce9e0911265ac6e9d5384bf096c5364ad62d8303ae76c4db231b667e" exitCode=0 Jul 24 03:18:09.351361 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:09.350854 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rw2n2" event={"ID":"5b31cd8e-39ec-4f6f-80ee-3168270efc37","Type":"ContainerDied","Data":"434e57b2ce9e0911265ac6e9d5384bf096c5364ad62d8303ae76c4db231b667e"} Jul 24 03:18:09.692524 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:09.692489 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls\") pod \"dns-default-4zq5x\" (UID: \"1a36f9a1-1cc8-4858-be3f-ec11e34eebab\") " pod="openshift-dns/dns-default-4zq5x" Jul 24 03:18:09.692719 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:09.692631 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 03:18:09.692719 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:09.692711 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls podName:1a36f9a1-1cc8-4858-be3f-ec11e34eebab nodeName:}" failed. No retries permitted until 2026-07-24 03:18:13.692689514 +0000 UTC m=+43.132108957 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls") pod "dns-default-4zq5x" (UID: "1a36f9a1-1cc8-4858-be3f-ec11e34eebab") : secret "dns-default-metrics-tls" not found Jul 24 03:18:10.360946 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:10.360783 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rw2n2" event={"ID":"5b31cd8e-39ec-4f6f-80ee-3168270efc37","Type":"ContainerStarted","Data":"589a4e46f48435d9222eb3a4606e6523f23ebf07254e56346ed73d1ffe02123c"} Jul 24 03:18:10.383060 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:10.382991 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rw2n2" podStartSLOduration=8.995902363999999 podStartE2EDuration="39.382977177s" totalImagesPullingTime="30.387074813s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 03:17:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:18:10.381757468 +0000 UTC m=+39.821176930" watchObservedRunningTime="2026-07-24 03:18:10.382977177 +0000 UTC m=+39.822396637" Jul 24 03:18:10.504519 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:10.504485 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:18:10.557875 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:10.557788 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:18:11.363992 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:11.363956 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-75ljs" event={"ID":"e63f0954-7da9-4fdc-a43d-d1cc3323aaa1","Type":"ContainerStarted","Data":"e301415c673775790d12aede58ee73abc10664343ec200a889524286231e79d2"} Jul 24 03:18:11.382551 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:11.382505 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-75ljs" podStartSLOduration=3.721888094 podStartE2EDuration="6.382487803s" totalImagesPullingTime="2.660599709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:18:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:18:11.38203895 +0000 UTC m=+40.821458411" watchObservedRunningTime="2026-07-24 03:18:11.382487803 +0000 UTC m=+40.821907262" Jul 24 03:18:13.722290 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:13.722252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls\") pod \"dns-default-4zq5x\" (UID: \"1a36f9a1-1cc8-4858-be3f-ec11e34eebab\") " pod="openshift-dns/dns-default-4zq5x" Jul 24 03:18:13.722733 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:13.722408 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 03:18:13.722733 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:13.722509 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls podName:1a36f9a1-1cc8-4858-be3f-ec11e34eebab nodeName:}" failed. No retries permitted until 2026-07-24 03:18:21.722491625 +0000 UTC m=+51.161911063 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls") pod "dns-default-4zq5x" (UID: "1a36f9a1-1cc8-4858-be3f-ec11e34eebab") : secret "dns-default-metrics-tls" not found Jul 24 03:18:14.017179 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:14.017095 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:18:14.070771 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:14.070732 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:18:21.770008 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:21.769968 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls\") pod \"dns-default-4zq5x\" (UID: \"1a36f9a1-1cc8-4858-be3f-ec11e34eebab\") " pod="openshift-dns/dns-default-4zq5x" Jul 24 03:18:21.770512 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:21.770113 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 03:18:21.770512 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:21.770187 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls podName:1a36f9a1-1cc8-4858-be3f-ec11e34eebab nodeName:}" failed. No retries permitted until 2026-07-24 03:18:37.770171104 +0000 UTC m=+67.209590542 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls") pod "dns-default-4zq5x" (UID: "1a36f9a1-1cc8-4858-be3f-ec11e34eebab") : secret "dns-default-metrics-tls" not found Jul 24 03:18:26.116120 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:26.116084 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:18:26.194361 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:26.194331 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:18:32.337591 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:32.337562 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sfxh7" Jul 24 03:18:32.478622 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:32.478596 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:18:32.546612 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:32.546579 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:18:36.671297 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:36.671254 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktg2b\" (UniqueName: \"kubernetes.io/projected/cb391e83-8f0f-424a-9f25-8b4a0ecf38d2-kube-api-access-ktg2b\") pod \"network-check-target-l4dwr\" (UID: \"cb391e83-8f0f-424a-9f25-8b4a0ecf38d2\") " pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:18:36.671297 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:36.671301 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad666805-ba79-4b67-8a3b-b9ba32afe705-original-pull-secret\") pod \"global-pull-secret-syncer-fhm9h\" (UID: \"ad666805-ba79-4b67-8a3b-b9ba32afe705\") " pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:18:36.671739 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:36.671323 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs\") pod \"network-metrics-daemon-8xwcn\" (UID: \"8fdc01f1-f7a1-4395-bae7-be0c9d76b76c\") " pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:18:36.674973 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:36.674954 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 03:18:36.675542 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:36.675526 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 03:18:36.675583 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:36.675562 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 03:18:36.682412 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:36.682396 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 24 03:18:36.682491 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:36.682480 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs podName:8fdc01f1-f7a1-4395-bae7-be0c9d76b76c nodeName:}" failed. No retries permitted until 2026-07-24 03:19:40.682447584 +0000 UTC m=+130.121867026 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs") pod "network-metrics-daemon-8xwcn" (UID: "8fdc01f1-f7a1-4395-bae7-be0c9d76b76c") : secret "metrics-daemon-secret" not found Jul 24 03:18:36.684642 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:36.684615 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 03:18:36.686254 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:36.686233 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ad666805-ba79-4b67-8a3b-b9ba32afe705-original-pull-secret\") pod \"global-pull-secret-syncer-fhm9h\" (UID: \"ad666805-ba79-4b67-8a3b-b9ba32afe705\") " pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:18:36.694718 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:36.694700 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktg2b\" (UniqueName: \"kubernetes.io/projected/cb391e83-8f0f-424a-9f25-8b4a0ecf38d2-kube-api-access-ktg2b\") pod \"network-check-target-l4dwr\" (UID: \"cb391e83-8f0f-424a-9f25-8b4a0ecf38d2\") " pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:18:36.739573 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:36.739541 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fhm9h" Jul 24 03:18:36.748768 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:36.748685 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n6kfx\"" Jul 24 03:18:36.756844 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:36.756821 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:18:36.870273 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:36.870244 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fhm9h"] Jul 24 03:18:36.873544 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:18:36.873523 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad666805_ba79_4b67_8a3b_b9ba32afe705.slice/crio-35736ca271108e042eb6c243592676fd1f177f5f65b9b67dc430b47425cbbaac WatchSource:0}: Error finding container 35736ca271108e042eb6c243592676fd1f177f5f65b9b67dc430b47425cbbaac: Status 404 returned error can't find the container with id 35736ca271108e042eb6c243592676fd1f177f5f65b9b67dc430b47425cbbaac Jul 24 03:18:36.881449 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:36.881429 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l4dwr"] Jul 24 03:18:36.884602 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:18:36.884583 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb391e83_8f0f_424a_9f25_8b4a0ecf38d2.slice/crio-eef3a650d418f9739db0153cf2af5dff6a5f7510eb0cc7ef5f6954a4a1f76141 WatchSource:0}: Error finding container eef3a650d418f9739db0153cf2af5dff6a5f7510eb0cc7ef5f6954a4a1f76141: Status 404 returned error can't find the container with id eef3a650d418f9739db0153cf2af5dff6a5f7510eb0cc7ef5f6954a4a1f76141 Jul 24 03:18:37.423152 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:37.423112 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l4dwr" event={"ID":"cb391e83-8f0f-424a-9f25-8b4a0ecf38d2","Type":"ContainerStarted","Data":"eef3a650d418f9739db0153cf2af5dff6a5f7510eb0cc7ef5f6954a4a1f76141"} Jul 24 03:18:37.424475 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:37.424432 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fhm9h" event={"ID":"ad666805-ba79-4b67-8a3b-b9ba32afe705","Type":"ContainerStarted","Data":"35736ca271108e042eb6c243592676fd1f177f5f65b9b67dc430b47425cbbaac"} Jul 24 03:18:37.777578 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:37.777485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls\") pod \"dns-default-4zq5x\" (UID: \"1a36f9a1-1cc8-4858-be3f-ec11e34eebab\") " pod="openshift-dns/dns-default-4zq5x" Jul 24 03:18:37.777989 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:37.777637 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 03:18:37.777989 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:18:37.777699 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls podName:1a36f9a1-1cc8-4858-be3f-ec11e34eebab nodeName:}" failed. No retries permitted until 2026-07-24 03:19:09.777681446 +0000 UTC m=+99.217100884 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls") pod "dns-default-4zq5x" (UID: "1a36f9a1-1cc8-4858-be3f-ec11e34eebab") : secret "dns-default-metrics-tls" not found Jul 24 03:18:42.439830 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:42.439793 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l4dwr" event={"ID":"cb391e83-8f0f-424a-9f25-8b4a0ecf38d2","Type":"ContainerStarted","Data":"c435f67587930c78fe9152de1470cbc3d5cf99198e476a78aca5c9f4b9025391"} Jul 24 03:18:42.440262 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:42.439929 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:18:42.441377 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:42.441358 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fhm9h" event={"ID":"ad666805-ba79-4b67-8a3b-b9ba32afe705","Type":"ContainerStarted","Data":"c6b95d252b731a34a50cdb6e6c5d383c1bc0a459161c0750b1e911e855106eb7"} Jul 24 03:18:42.460714 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:18:42.460672 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-l4dwr" podStartSLOduration=66.686751846 podStartE2EDuration="1m11.460661863s" totalImagesPullingTime="4.773910017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:18:42.460179863 +0000 UTC m=+71.899599333" watchObservedRunningTime="2026-07-24 03:18:42.460661863 +0000 UTC m=+71.900081323" Jul 24 03:19:01.060874 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.060735 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fhm9h" podStartSLOduration=85.272172274 podStartE2EDuration="1m30.060719868s" totalImagesPullingTime="4.788547594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 03:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:18:42.479823112 +0000 UTC m=+71.919242572" watchObservedRunningTime="2026-07-24 03:19:01.060719868 +0000 UTC m=+90.500139306" Jul 24 03:19:01.061759 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.061044 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64bc64b8b6-fschj"] Jul 24 03:19:01.065178 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.065158 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64bc64b8b6-fschj" Jul 24 03:19:01.068677 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.068655 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 24 03:19:01.068771 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.068680 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 24 03:19:01.068771 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.068719 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-pddqf\"" Jul 24 03:19:01.068771 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.068729 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 24 03:19:01.068871 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.068658 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 24 03:19:01.073124 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.073105 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64bc64b8b6-fschj"] Jul 24 03:19:01.096263 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.096233 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8fdf87fc4-f464k"] Jul 24 03:19:01.098943 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.098928 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8fdf87fc4-f464k" Jul 24 03:19:01.101333 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.101318 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 24 03:19:01.106244 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.106225 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8fdf87fc4-f464k"] Jul 24 03:19:01.138069 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.138039 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vf6k\" (UniqueName: \"kubernetes.io/projected/fbe14f31-fb41-46e7-a3b4-e84af9d5ab46-kube-api-access-9vf6k\") pod \"klusterlet-addon-workmgr-8fdf87fc4-f464k\" (UID: \"fbe14f31-fb41-46e7-a3b4-e84af9d5ab46\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8fdf87fc4-f464k" Jul 24 03:19:01.138215 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.138087 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50564691-a25e-44b1-a346-4131ad26e1f2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-64bc64b8b6-fschj\" (UID: \"50564691-a25e-44b1-a346-4131ad26e1f2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64bc64b8b6-fschj" Jul 24 03:19:01.138215 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.138106 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbe14f31-fb41-46e7-a3b4-e84af9d5ab46-tmp\") pod \"klusterlet-addon-workmgr-8fdf87fc4-f464k\" (UID: \"fbe14f31-fb41-46e7-a3b4-e84af9d5ab46\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8fdf87fc4-f464k" Jul 24 03:19:01.138215 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.138174 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fbe14f31-fb41-46e7-a3b4-e84af9d5ab46-klusterlet-config\") pod \"klusterlet-addon-workmgr-8fdf87fc4-f464k\" (UID: \"fbe14f31-fb41-46e7-a3b4-e84af9d5ab46\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8fdf87fc4-f464k" Jul 24 03:19:01.138215 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.138203 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6rrc\" (UniqueName: \"kubernetes.io/projected/50564691-a25e-44b1-a346-4131ad26e1f2-kube-api-access-s6rrc\") pod \"managed-serviceaccount-addon-agent-64bc64b8b6-fschj\" (UID: \"50564691-a25e-44b1-a346-4131ad26e1f2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64bc64b8b6-fschj" Jul 24 03:19:01.239278 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.239235 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fbe14f31-fb41-46e7-a3b4-e84af9d5ab46-klusterlet-config\") pod \"klusterlet-addon-workmgr-8fdf87fc4-f464k\" (UID: \"fbe14f31-fb41-46e7-a3b4-e84af9d5ab46\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8fdf87fc4-f464k" Jul 24 03:19:01.239278 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.239277 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6rrc\" (UniqueName: \"kubernetes.io/projected/50564691-a25e-44b1-a346-4131ad26e1f2-kube-api-access-s6rrc\") pod \"managed-serviceaccount-addon-agent-64bc64b8b6-fschj\" (UID: \"50564691-a25e-44b1-a346-4131ad26e1f2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64bc64b8b6-fschj" Jul 24 03:19:01.239561 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.239318 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vf6k\" (UniqueName: \"kubernetes.io/projected/fbe14f31-fb41-46e7-a3b4-e84af9d5ab46-kube-api-access-9vf6k\") pod \"klusterlet-addon-workmgr-8fdf87fc4-f464k\" (UID: \"fbe14f31-fb41-46e7-a3b4-e84af9d5ab46\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8fdf87fc4-f464k" Jul 24 03:19:01.239561 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.239534 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50564691-a25e-44b1-a346-4131ad26e1f2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-64bc64b8b6-fschj\" (UID: \"50564691-a25e-44b1-a346-4131ad26e1f2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64bc64b8b6-fschj" Jul 24 03:19:01.239561 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.239560 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbe14f31-fb41-46e7-a3b4-e84af9d5ab46-tmp\") pod \"klusterlet-addon-workmgr-8fdf87fc4-f464k\" (UID: \"fbe14f31-fb41-46e7-a3b4-e84af9d5ab46\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8fdf87fc4-f464k" Jul 24 03:19:01.239879 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.239863 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbe14f31-fb41-46e7-a3b4-e84af9d5ab46-tmp\") pod \"klusterlet-addon-workmgr-8fdf87fc4-f464k\" (UID: \"fbe14f31-fb41-46e7-a3b4-e84af9d5ab46\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8fdf87fc4-f464k" Jul 24 03:19:01.241949 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.241915 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fbe14f31-fb41-46e7-a3b4-e84af9d5ab46-klusterlet-config\") pod \"klusterlet-addon-workmgr-8fdf87fc4-f464k\" (UID: \"fbe14f31-fb41-46e7-a3b4-e84af9d5ab46\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8fdf87fc4-f464k" Jul 24 03:19:01.242056 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.241963 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50564691-a25e-44b1-a346-4131ad26e1f2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-64bc64b8b6-fschj\" (UID: \"50564691-a25e-44b1-a346-4131ad26e1f2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64bc64b8b6-fschj" Jul 24 03:19:01.247953 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.247926 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6rrc\" (UniqueName: \"kubernetes.io/projected/50564691-a25e-44b1-a346-4131ad26e1f2-kube-api-access-s6rrc\") pod \"managed-serviceaccount-addon-agent-64bc64b8b6-fschj\" (UID: \"50564691-a25e-44b1-a346-4131ad26e1f2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64bc64b8b6-fschj" Jul 24 03:19:01.248761 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.248743 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vf6k\" (UniqueName: \"kubernetes.io/projected/fbe14f31-fb41-46e7-a3b4-e84af9d5ab46-kube-api-access-9vf6k\") pod \"klusterlet-addon-workmgr-8fdf87fc4-f464k\" (UID: \"fbe14f31-fb41-46e7-a3b4-e84af9d5ab46\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8fdf87fc4-f464k" Jul 24 03:19:01.385634 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.385602 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64bc64b8b6-fschj" Jul 24 03:19:01.408916 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.408882 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8fdf87fc4-f464k" Jul 24 03:19:01.515837 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.515806 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64bc64b8b6-fschj"] Jul 24 03:19:01.518701 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:19:01.518675 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50564691_a25e_44b1_a346_4131ad26e1f2.slice/crio-9888dc421f9fa635997a9b17ec1c8eb622bb1d054b8c6417bd3da8f740326119 WatchSource:0}: Error finding container 9888dc421f9fa635997a9b17ec1c8eb622bb1d054b8c6417bd3da8f740326119: Status 404 returned error can't find the container with id 9888dc421f9fa635997a9b17ec1c8eb622bb1d054b8c6417bd3da8f740326119 Jul 24 03:19:01.535997 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:01.535974 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8fdf87fc4-f464k"] Jul 24 03:19:01.539245 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:19:01.539224 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe14f31_fb41_46e7_a3b4_e84af9d5ab46.slice/crio-ca42d458162d31a0eb0a5f11f9f6f3ed5f01e4289f3e8115b52c1a774a5ce61d WatchSource:0}: Error finding container ca42d458162d31a0eb0a5f11f9f6f3ed5f01e4289f3e8115b52c1a774a5ce61d: Status 404 returned error can't find the container with id ca42d458162d31a0eb0a5f11f9f6f3ed5f01e4289f3e8115b52c1a774a5ce61d Jul 24 03:19:02.492174 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:02.492130 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8fdf87fc4-f464k" event={"ID":"fbe14f31-fb41-46e7-a3b4-e84af9d5ab46","Type":"ContainerStarted","Data":"ca42d458162d31a0eb0a5f11f9f6f3ed5f01e4289f3e8115b52c1a774a5ce61d"} Jul 24 03:19:02.493843 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:02.493814 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64bc64b8b6-fschj" event={"ID":"50564691-a25e-44b1-a346-4131ad26e1f2","Type":"ContainerStarted","Data":"9888dc421f9fa635997a9b17ec1c8eb622bb1d054b8c6417bd3da8f740326119"} Jul 24 03:19:05.503703 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:05.503664 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64bc64b8b6-fschj" event={"ID":"50564691-a25e-44b1-a346-4131ad26e1f2","Type":"ContainerStarted","Data":"ff5247330bc8b1c6dce8acfeb1211faa824d7da6061c62bbdd3fd9a9f762f4a1"} Jul 24 03:19:05.505047 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:05.505021 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8fdf87fc4-f464k" event={"ID":"fbe14f31-fb41-46e7-a3b4-e84af9d5ab46","Type":"ContainerStarted","Data":"fb64df6f7f37ffcf336f0d8e13b0f9f3b168bd7c30c9d64e1319bc6b6971604c"} Jul 24 03:19:05.505256 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:05.505241 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8fdf87fc4-f464k" Jul 24 03:19:05.506676 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:05.506650 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8fdf87fc4-f464k" Jul 24 03:19:05.519902 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:05.519850 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64bc64b8b6-fschj" podStartSLOduration=0.741471847 podStartE2EDuration="4.51983637s" totalImagesPullingTime="3.778364523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:19:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:19:05.519186695 +0000 UTC m=+94.958606156" watchObservedRunningTime="2026-07-24 03:19:05.51983637 +0000 UTC m=+94.959255823" Jul 24 03:19:05.536541 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:05.536494 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8fdf87fc4-f464k" podStartSLOduration=0.766835989 podStartE2EDuration="4.536481871s" totalImagesPullingTime="3.769645882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:19:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:19:05.535840356 +0000 UTC m=+94.975259816" watchObservedRunningTime="2026-07-24 03:19:05.536481871 +0000 UTC m=+94.975901322" Jul 24 03:19:09.801762 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:09.801723 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls\") pod \"dns-default-4zq5x\" (UID: \"1a36f9a1-1cc8-4858-be3f-ec11e34eebab\") " pod="openshift-dns/dns-default-4zq5x" Jul 24 03:19:09.802211 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:19:09.801863 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 03:19:09.802211 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:19:09.801924 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls podName:1a36f9a1-1cc8-4858-be3f-ec11e34eebab nodeName:}" failed. No retries permitted until 2026-07-24 03:20:13.801908861 +0000 UTC m=+163.241328299 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls") pod "dns-default-4zq5x" (UID: "1a36f9a1-1cc8-4858-be3f-ec11e34eebab") : secret "dns-default-metrics-tls" not found Jul 24 03:19:13.446406 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:13.446373 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-l4dwr" Jul 24 03:19:40.717099 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:19:40.717048 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs\") pod \"network-metrics-daemon-8xwcn\" (UID: \"8fdc01f1-f7a1-4395-bae7-be0c9d76b76c\") " pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:19:40.717603 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:19:40.717207 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 24 03:19:40.717603 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:19:40.717282 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs podName:8fdc01f1-f7a1-4395-bae7-be0c9d76b76c nodeName:}" failed. No retries permitted until 2026-07-24 03:21:42.71726391 +0000 UTC m=+252.156683364 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs") pod "network-metrics-daemon-8xwcn" (UID: "8fdc01f1-f7a1-4395-bae7-be0c9d76b76c") : secret "metrics-daemon-secret" not found Jul 24 03:20:08.978762 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:20:08.978705 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-4zq5x" podUID="1a36f9a1-1cc8-4858-be3f-ec11e34eebab" Jul 24 03:20:09.662146 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:20:09.662119 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4zq5x" Jul 24 03:20:10.052067 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:20:10.051964 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-8xwcn" podUID="8fdc01f1-f7a1-4395-bae7-be0c9d76b76c" Jul 24 03:20:13.846978 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:20:13.846930 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls\") pod \"dns-default-4zq5x\" (UID: \"1a36f9a1-1cc8-4858-be3f-ec11e34eebab\") " pod="openshift-dns/dns-default-4zq5x" Jul 24 03:20:13.847369 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:20:13.847077 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 03:20:13.847369 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:20:13.847141 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls podName:1a36f9a1-1cc8-4858-be3f-ec11e34eebab nodeName:}" failed. No retries permitted until 2026-07-24 03:22:15.847126105 +0000 UTC m=+285.286545543 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls") pod "dns-default-4zq5x" (UID: "1a36f9a1-1cc8-4858-be3f-ec11e34eebab") : secret "dns-default-metrics-tls" not found Jul 24 03:20:24.027230 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:20:24.027151 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:20:32.231869 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:20:32.231834 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:20:52.885609 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:20:52.885576 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:05.506160 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:05.506107 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8fdf87fc4-f464k" podUID="fbe14f31-fb41-46e7-a3b4-e84af9d5ab46" containerName="acm-agent" probeResult="failure" output="Get \"http://10.134.0.9:8000/readyz\": dial tcp 10.134.0.9:8000: connect: connection refused" Jul 24 03:21:05.796474 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:05.796379 2579 generic.go:356] "Generic (PLEG): container finished" podID="50564691-a25e-44b1-a346-4131ad26e1f2" containerID="ff5247330bc8b1c6dce8acfeb1211faa824d7da6061c62bbdd3fd9a9f762f4a1" exitCode=255 Jul 24 03:21:05.796474 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:05.796454 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64bc64b8b6-fschj" event={"ID":"50564691-a25e-44b1-a346-4131ad26e1f2","Type":"ContainerDied","Data":"ff5247330bc8b1c6dce8acfeb1211faa824d7da6061c62bbdd3fd9a9f762f4a1"} Jul 24 03:21:05.798012 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:05.797986 2579 generic.go:356] "Generic (PLEG): container finished" podID="fbe14f31-fb41-46e7-a3b4-e84af9d5ab46" containerID="fb64df6f7f37ffcf336f0d8e13b0f9f3b168bd7c30c9d64e1319bc6b6971604c" exitCode=1 Jul 24 03:21:05.798116 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:05.798041 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8fdf87fc4-f464k" event={"ID":"fbe14f31-fb41-46e7-a3b4-e84af9d5ab46","Type":"ContainerDied","Data":"fb64df6f7f37ffcf336f0d8e13b0f9f3b168bd7c30c9d64e1319bc6b6971604c"} Jul 24 03:21:05.798374 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:05.798359 2579 scope.go:117] "RemoveContainer" containerID="fb64df6f7f37ffcf336f0d8e13b0f9f3b168bd7c30c9d64e1319bc6b6971604c" Jul 24 03:21:05.802987 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:05.802961 2579 scope.go:117] "RemoveContainer" containerID="ff5247330bc8b1c6dce8acfeb1211faa824d7da6061c62bbdd3fd9a9f762f4a1" Jul 24 03:21:06.803625 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:06.803595 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8fdf87fc4-f464k" event={"ID":"fbe14f31-fb41-46e7-a3b4-e84af9d5ab46","Type":"ContainerStarted","Data":"0a2ac0ee0af6371d57e29342d477c692d44cf28ed0b0c538afaaa19478d4cec6"} Jul 24 03:21:06.804089 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:06.803904 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8fdf87fc4-f464k" Jul 24 03:21:06.804550 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:06.804528 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8fdf87fc4-f464k" Jul 24 03:21:06.805296 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:06.805278 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64bc64b8b6-fschj" event={"ID":"50564691-a25e-44b1-a346-4131ad26e1f2","Type":"ContainerStarted","Data":"9d9c8cd86a8d8e798287e7288fc3c16808f8e74d21db6a0c0bdeabfef890930a"} Jul 24 03:21:18.803785 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.803753 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q5l2j"] Jul 24 03:21:18.806914 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.806895 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q5l2j" Jul 24 03:21:18.809225 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.809201 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 24 03:21:18.809340 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.809234 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 24 03:21:18.809340 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.809264 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 24 03:21:18.809540 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.809521 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 24 03:21:18.810220 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.810204 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-fj98p\"" Jul 24 03:21:18.811281 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.811262 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-8gpjf"] Jul 24 03:21:18.814015 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.813999 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-8gpjf" Jul 24 03:21:18.816342 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.816326 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-c2svp\"" Jul 24 03:21:18.816599 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.816585 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 24 03:21:18.816681 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.816590 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 24 03:21:18.816945 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.816924 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 24 03:21:18.817182 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.817167 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 24 03:21:18.819951 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.819930 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q5l2j"] Jul 24 03:21:18.822892 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.822875 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 24 03:21:18.828293 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.828275 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-8gpjf"] Jul 24 03:21:18.874780 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.874757 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr8rv\" (UniqueName: \"kubernetes.io/projected/0d716e5b-e403-4cf6-94f4-8dbca03f5e25-kube-api-access-jr8rv\") pod \"service-ca-operator-5d6c75f84f-q5l2j\" (UID: \"0d716e5b-e403-4cf6-94f4-8dbca03f5e25\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q5l2j" Jul 24 03:21:18.874780 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.874784 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/587fc069-a2aa-4701-82e9-65e01ab74ade-tmp\") pod \"insights-operator-5885fdb759-8gpjf\" (UID: \"587fc069-a2aa-4701-82e9-65e01ab74ade\") " pod="openshift-insights/insights-operator-5885fdb759-8gpjf" Jul 24 03:21:18.874951 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.874808 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0d716e5b-e403-4cf6-94f4-8dbca03f5e25-config\") pod \"service-ca-operator-5d6c75f84f-q5l2j\" (UID: \"0d716e5b-e403-4cf6-94f4-8dbca03f5e25\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q5l2j" Jul 24 03:21:18.874951 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.874839 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgqvl\" (UniqueName: \"kubernetes.io/projected/587fc069-a2aa-4701-82e9-65e01ab74ade-kube-api-access-tgqvl\") pod \"insights-operator-5885fdb759-8gpjf\" (UID: \"587fc069-a2aa-4701-82e9-65e01ab74ade\") " pod="openshift-insights/insights-operator-5885fdb759-8gpjf" Jul 24 03:21:18.874951 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.874869 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0d716e5b-e403-4cf6-94f4-8dbca03f5e25-serving-cert\") pod \"service-ca-operator-5d6c75f84f-q5l2j\" (UID: \"0d716e5b-e403-4cf6-94f4-8dbca03f5e25\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q5l2j" Jul 24 03:21:18.874951 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.874884 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/587fc069-a2aa-4701-82e9-65e01ab74ade-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-8gpjf\" (UID: \"587fc069-a2aa-4701-82e9-65e01ab74ade\") " pod="openshift-insights/insights-operator-5885fdb759-8gpjf" Jul 24 03:21:18.874951 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.874907 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/587fc069-a2aa-4701-82e9-65e01ab74ade-serving-cert\") pod \"insights-operator-5885fdb759-8gpjf\" (UID: \"587fc069-a2aa-4701-82e9-65e01ab74ade\") " pod="openshift-insights/insights-operator-5885fdb759-8gpjf" Jul 24 03:21:18.875153 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.874958 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/587fc069-a2aa-4701-82e9-65e01ab74ade-snapshots\") pod \"insights-operator-5885fdb759-8gpjf\" (UID: \"587fc069-a2aa-4701-82e9-65e01ab74ade\") " pod="openshift-insights/insights-operator-5885fdb759-8gpjf" Jul 24 03:21:18.875153 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.874995 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/587fc069-a2aa-4701-82e9-65e01ab74ade-service-ca-bundle\") pod \"insights-operator-5885fdb759-8gpjf\" (UID: \"587fc069-a2aa-4701-82e9-65e01ab74ade\") " pod="openshift-insights/insights-operator-5885fdb759-8gpjf" Jul 24 03:21:18.931647 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.931614 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:18.975699 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.975675 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0d716e5b-e403-4cf6-94f4-8dbca03f5e25-serving-cert\") pod \"service-ca-operator-5d6c75f84f-q5l2j\" (UID: \"0d716e5b-e403-4cf6-94f4-8dbca03f5e25\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q5l2j" Jul 24 03:21:18.975797 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.975703 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/587fc069-a2aa-4701-82e9-65e01ab74ade-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-8gpjf\" (UID: \"587fc069-a2aa-4701-82e9-65e01ab74ade\") " pod="openshift-insights/insights-operator-5885fdb759-8gpjf" Jul 24 03:21:18.975797 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.975726 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/587fc069-a2aa-4701-82e9-65e01ab74ade-serving-cert\") pod \"insights-operator-5885fdb759-8gpjf\" (UID: \"587fc069-a2aa-4701-82e9-65e01ab74ade\") " pod="openshift-insights/insights-operator-5885fdb759-8gpjf" Jul 24 03:21:18.975797 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.975744 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/587fc069-a2aa-4701-82e9-65e01ab74ade-snapshots\") pod \"insights-operator-5885fdb759-8gpjf\" (UID: \"587fc069-a2aa-4701-82e9-65e01ab74ade\") " pod="openshift-insights/insights-operator-5885fdb759-8gpjf" Jul 24 03:21:18.975950 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.975791 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/587fc069-a2aa-4701-82e9-65e01ab74ade-service-ca-bundle\") pod \"insights-operator-5885fdb759-8gpjf\" (UID: \"587fc069-a2aa-4701-82e9-65e01ab74ade\") " pod="openshift-insights/insights-operator-5885fdb759-8gpjf" Jul 24 03:21:18.975950 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.975892 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr8rv\" (UniqueName: \"kubernetes.io/projected/0d716e5b-e403-4cf6-94f4-8dbca03f5e25-kube-api-access-jr8rv\") pod \"service-ca-operator-5d6c75f84f-q5l2j\" (UID: \"0d716e5b-e403-4cf6-94f4-8dbca03f5e25\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q5l2j" Jul 24 03:21:18.975950 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.975920 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/587fc069-a2aa-4701-82e9-65e01ab74ade-tmp\") pod \"insights-operator-5885fdb759-8gpjf\" (UID: \"587fc069-a2aa-4701-82e9-65e01ab74ade\") " pod="openshift-insights/insights-operator-5885fdb759-8gpjf" Jul 24 03:21:18.976097 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.975957 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0d716e5b-e403-4cf6-94f4-8dbca03f5e25-config\") pod \"service-ca-operator-5d6c75f84f-q5l2j\" (UID: \"0d716e5b-e403-4cf6-94f4-8dbca03f5e25\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q5l2j" Jul 24 03:21:18.976097 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.976008 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgqvl\" (UniqueName: \"kubernetes.io/projected/587fc069-a2aa-4701-82e9-65e01ab74ade-kube-api-access-tgqvl\") pod \"insights-operator-5885fdb759-8gpjf\" (UID: \"587fc069-a2aa-4701-82e9-65e01ab74ade\") " pod="openshift-insights/insights-operator-5885fdb759-8gpjf" Jul 24 03:21:18.976512 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.976488 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/587fc069-a2aa-4701-82e9-65e01ab74ade-tmp\") pod \"insights-operator-5885fdb759-8gpjf\" (UID: \"587fc069-a2aa-4701-82e9-65e01ab74ade\") " pod="openshift-insights/insights-operator-5885fdb759-8gpjf" Jul 24 03:21:18.976612 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.976497 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0d716e5b-e403-4cf6-94f4-8dbca03f5e25-config\") pod \"service-ca-operator-5d6c75f84f-q5l2j\" (UID: \"0d716e5b-e403-4cf6-94f4-8dbca03f5e25\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q5l2j" Jul 24 03:21:18.976612 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.976534 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/587fc069-a2aa-4701-82e9-65e01ab74ade-snapshots\") pod \"insights-operator-5885fdb759-8gpjf\" (UID: \"587fc069-a2aa-4701-82e9-65e01ab74ade\") " pod="openshift-insights/insights-operator-5885fdb759-8gpjf" Jul 24 03:21:18.976853 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.976838 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/587fc069-a2aa-4701-82e9-65e01ab74ade-service-ca-bundle\") pod \"insights-operator-5885fdb759-8gpjf\" (UID: \"587fc069-a2aa-4701-82e9-65e01ab74ade\") " pod="openshift-insights/insights-operator-5885fdb759-8gpjf" Jul 24 03:21:18.977298 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.977278 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/587fc069-a2aa-4701-82e9-65e01ab74ade-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-8gpjf\" (UID: \"587fc069-a2aa-4701-82e9-65e01ab74ade\") " pod="openshift-insights/insights-operator-5885fdb759-8gpjf" Jul 24 03:21:18.978255 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.978231 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0d716e5b-e403-4cf6-94f4-8dbca03f5e25-serving-cert\") pod \"service-ca-operator-5d6c75f84f-q5l2j\" (UID: \"0d716e5b-e403-4cf6-94f4-8dbca03f5e25\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q5l2j" Jul 24 03:21:18.978336 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.978270 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/587fc069-a2aa-4701-82e9-65e01ab74ade-serving-cert\") pod \"insights-operator-5885fdb759-8gpjf\" (UID: \"587fc069-a2aa-4701-82e9-65e01ab74ade\") " pod="openshift-insights/insights-operator-5885fdb759-8gpjf" Jul 24 03:21:18.983118 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.983080 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:18.983434 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.983407 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgqvl\" (UniqueName: \"kubernetes.io/projected/587fc069-a2aa-4701-82e9-65e01ab74ade-kube-api-access-tgqvl\") pod \"insights-operator-5885fdb759-8gpjf\" (UID: \"587fc069-a2aa-4701-82e9-65e01ab74ade\") " pod="openshift-insights/insights-operator-5885fdb759-8gpjf" Jul 24 03:21:18.983576 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:18.983557 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr8rv\" (UniqueName: \"kubernetes.io/projected/0d716e5b-e403-4cf6-94f4-8dbca03f5e25-kube-api-access-jr8rv\") pod \"service-ca-operator-5d6c75f84f-q5l2j\" (UID: \"0d716e5b-e403-4cf6-94f4-8dbca03f5e25\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q5l2j" Jul 24 03:21:19.115905 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:19.115873 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q5l2j" Jul 24 03:21:19.124056 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:19.124030 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-8gpjf" Jul 24 03:21:19.247412 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:19.247385 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q5l2j"] Jul 24 03:21:19.251657 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:21:19.251629 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d716e5b_e403_4cf6_94f4_8dbca03f5e25.slice/crio-99b7d728e29bcaf8e9f0a96c948e61386cb48823b383d46152509cfececd8f29 WatchSource:0}: Error finding container 99b7d728e29bcaf8e9f0a96c948e61386cb48823b383d46152509cfececd8f29: Status 404 returned error can't find the container with id 99b7d728e29bcaf8e9f0a96c948e61386cb48823b383d46152509cfececd8f29 Jul 24 03:21:19.266118 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:19.266094 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-8gpjf"] Jul 24 03:21:19.271207 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:21:19.271183 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587fc069_a2aa_4701_82e9_65e01ab74ade.slice/crio-cfa1c390eb48e00d4eabd66e123d72c02cc49fd5a00f820cb3a74563dbb18a24 WatchSource:0}: Error finding container cfa1c390eb48e00d4eabd66e123d72c02cc49fd5a00f820cb3a74563dbb18a24: Status 404 returned error can't find the container with id cfa1c390eb48e00d4eabd66e123d72c02cc49fd5a00f820cb3a74563dbb18a24 Jul 24 03:21:19.847771 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:19.847737 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q5l2j" event={"ID":"0d716e5b-e403-4cf6-94f4-8dbca03f5e25","Type":"ContainerStarted","Data":"99b7d728e29bcaf8e9f0a96c948e61386cb48823b383d46152509cfececd8f29"} Jul 24 03:21:19.849032 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:19.848997 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-8gpjf" event={"ID":"587fc069-a2aa-4701-82e9-65e01ab74ade","Type":"ContainerStarted","Data":"cfa1c390eb48e00d4eabd66e123d72c02cc49fd5a00f820cb3a74563dbb18a24"} Jul 24 03:21:21.857151 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:21.857110 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q5l2j" event={"ID":"0d716e5b-e403-4cf6-94f4-8dbca03f5e25","Type":"ContainerStarted","Data":"7f311605b7a6a5b4c94fc2b1aff76a6802529c6b9ee7446331b280943cda6c1f"} Jul 24 03:21:21.858925 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:21.858897 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-8gpjf" event={"ID":"587fc069-a2aa-4701-82e9-65e01ab74ade","Type":"ContainerStarted","Data":"5cb9f94570650cf345e67830f0ea89eadbb21b54a3b05956840b4a2500d0cc93"} Jul 24 03:21:21.871108 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:21.871060 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q5l2j" podStartSLOduration=1.353830994 podStartE2EDuration="3.871045415s" totalImagesPullingTime="2.517214421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:21:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:21:21.869887876 +0000 UTC m=+231.309307336" watchObservedRunningTime="2026-07-24 03:21:21.871045415 +0000 UTC m=+231.310464875" Jul 24 03:21:21.886402 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:21.886353 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-8gpjf" podStartSLOduration=1.386854003 podStartE2EDuration="3.88633461s" totalImagesPullingTime="2.499480607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:21:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:21:21.885392841 +0000 UTC m=+231.324812298" watchObservedRunningTime="2026-07-24 03:21:21.88633461 +0000 UTC m=+231.325754071" Jul 24 03:21:22.593447 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:22.593415 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:22.648620 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:22.648588 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:22.906936 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:22.906907 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:22.941519 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:22.941481 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-tsp28"] Jul 24 03:21:22.945049 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:22.945022 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-tsp28" Jul 24 03:21:22.948583 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:22.948336 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 24 03:21:22.949139 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:22.949116 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tthjz\"" Jul 24 03:21:22.949139 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:22.949125 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 24 03:21:22.961622 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:22.961368 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xnpz8"] Jul 24 03:21:22.964810 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:22.964774 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-tsp28"] Jul 24 03:21:22.964937 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:22.964889 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xnpz8" Jul 24 03:21:22.966868 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:22.966847 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 24 03:21:22.967014 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:22.966996 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fs6xn\"" Jul 24 03:21:22.967149 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:22.967131 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 24 03:21:22.976484 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:22.976447 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xnpz8"] Jul 24 03:21:22.992693 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:22.992670 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:23.011288 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:23.011264 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhhdn\" (UniqueName: \"kubernetes.io/projected/12956bc1-3664-46a8-97e6-be81968535d9-kube-api-access-xhhdn\") pod \"migrator-6865c9967-tsp28\" (UID: \"12956bc1-3664-46a8-97e6-be81968535d9\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-tsp28" Jul 24 03:21:23.106919 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:23.106886 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:23.112039 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:23.112014 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9373ab5f-eb4b-495c-9f7f-b0f00e884339-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xnpz8\" (UID: \"9373ab5f-eb4b-495c-9f7f-b0f00e884339\") " pod="openshift-insights/insights-runtime-extractor-xnpz8" Jul 24 03:21:23.112170 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:23.112099 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhhdn\" (UniqueName: \"kubernetes.io/projected/12956bc1-3664-46a8-97e6-be81968535d9-kube-api-access-xhhdn\") pod \"migrator-6865c9967-tsp28\" (UID: \"12956bc1-3664-46a8-97e6-be81968535d9\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-tsp28" Jul 24 03:21:23.112210 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:23.112188 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9373ab5f-eb4b-495c-9f7f-b0f00e884339-crio-socket\") pod \"insights-runtime-extractor-xnpz8\" (UID: \"9373ab5f-eb4b-495c-9f7f-b0f00e884339\") " pod="openshift-insights/insights-runtime-extractor-xnpz8" Jul 24 03:21:23.112250 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:23.112214 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpxrn\" (UniqueName: \"kubernetes.io/projected/9373ab5f-eb4b-495c-9f7f-b0f00e884339-kube-api-access-zpxrn\") pod \"insights-runtime-extractor-xnpz8\" (UID: \"9373ab5f-eb4b-495c-9f7f-b0f00e884339\") " pod="openshift-insights/insights-runtime-extractor-xnpz8" Jul 24 03:21:23.112250 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:23.112243 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9373ab5f-eb4b-495c-9f7f-b0f00e884339-data-volume\") pod \"insights-runtime-extractor-xnpz8\" (UID: \"9373ab5f-eb4b-495c-9f7f-b0f00e884339\") " pod="openshift-insights/insights-runtime-extractor-xnpz8" Jul 24 03:21:23.112322 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:23.112263 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9373ab5f-eb4b-495c-9f7f-b0f00e884339-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xnpz8\" (UID: \"9373ab5f-eb4b-495c-9f7f-b0f00e884339\") " pod="openshift-insights/insights-runtime-extractor-xnpz8" Jul 24 03:21:23.119638 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:23.119616 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhhdn\" (UniqueName: \"kubernetes.io/projected/12956bc1-3664-46a8-97e6-be81968535d9-kube-api-access-xhhdn\") pod \"migrator-6865c9967-tsp28\" (UID: \"12956bc1-3664-46a8-97e6-be81968535d9\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-tsp28" Jul 24 03:21:23.213009 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:23.212932 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9373ab5f-eb4b-495c-9f7f-b0f00e884339-crio-socket\") pod \"insights-runtime-extractor-xnpz8\" (UID: \"9373ab5f-eb4b-495c-9f7f-b0f00e884339\") " pod="openshift-insights/insights-runtime-extractor-xnpz8" Jul 24 03:21:23.213009 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:23.212972 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpxrn\" (UniqueName: \"kubernetes.io/projected/9373ab5f-eb4b-495c-9f7f-b0f00e884339-kube-api-access-zpxrn\") pod \"insights-runtime-extractor-xnpz8\" (UID: \"9373ab5f-eb4b-495c-9f7f-b0f00e884339\") " pod="openshift-insights/insights-runtime-extractor-xnpz8" Jul 24 03:21:23.213009 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:23.213009 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9373ab5f-eb4b-495c-9f7f-b0f00e884339-data-volume\") pod \"insights-runtime-extractor-xnpz8\" (UID: \"9373ab5f-eb4b-495c-9f7f-b0f00e884339\") " pod="openshift-insights/insights-runtime-extractor-xnpz8" Jul 24 03:21:23.213254 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:23.213035 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9373ab5f-eb4b-495c-9f7f-b0f00e884339-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xnpz8\" (UID: \"9373ab5f-eb4b-495c-9f7f-b0f00e884339\") " pod="openshift-insights/insights-runtime-extractor-xnpz8" Jul 24 03:21:23.213254 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:23.213051 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9373ab5f-eb4b-495c-9f7f-b0f00e884339-crio-socket\") pod \"insights-runtime-extractor-xnpz8\" (UID: \"9373ab5f-eb4b-495c-9f7f-b0f00e884339\") " pod="openshift-insights/insights-runtime-extractor-xnpz8" Jul 24 03:21:23.213254 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:23.213066 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9373ab5f-eb4b-495c-9f7f-b0f00e884339-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xnpz8\" (UID: \"9373ab5f-eb4b-495c-9f7f-b0f00e884339\") " pod="openshift-insights/insights-runtime-extractor-xnpz8" Jul 24 03:21:23.213254 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:21:23.213124 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 03:21:23.213254 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:21:23.213174 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 03:21:23.213254 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:21:23.213204 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9373ab5f-eb4b-495c-9f7f-b0f00e884339-kube-rbac-proxy-cm podName:9373ab5f-eb4b-495c-9f7f-b0f00e884339 nodeName:}" failed. No retries permitted until 2026-07-24 03:21:23.713182828 +0000 UTC m=+233.152602273 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9373ab5f-eb4b-495c-9f7f-b0f00e884339-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xnpz8" (UID: "9373ab5f-eb4b-495c-9f7f-b0f00e884339") : configmap "kube-rbac-proxy" not found Jul 24 03:21:23.213254 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:21:23.213233 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9373ab5f-eb4b-495c-9f7f-b0f00e884339-insights-runtime-extractor-tls podName:9373ab5f-eb4b-495c-9f7f-b0f00e884339 nodeName:}" failed. No retries permitted until 2026-07-24 03:21:23.713214402 +0000 UTC m=+233.152633865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9373ab5f-eb4b-495c-9f7f-b0f00e884339-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xnpz8" (UID: "9373ab5f-eb4b-495c-9f7f-b0f00e884339") : secret "insights-runtime-extractor-tls" not found Jul 24 03:21:23.213569 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:23.213299 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9373ab5f-eb4b-495c-9f7f-b0f00e884339-data-volume\") pod \"insights-runtime-extractor-xnpz8\" (UID: \"9373ab5f-eb4b-495c-9f7f-b0f00e884339\") " pod="openshift-insights/insights-runtime-extractor-xnpz8" Jul 24 03:21:23.220978 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:23.220957 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpxrn\" (UniqueName: \"kubernetes.io/projected/9373ab5f-eb4b-495c-9f7f-b0f00e884339-kube-api-access-zpxrn\") pod \"insights-runtime-extractor-xnpz8\" (UID: \"9373ab5f-eb4b-495c-9f7f-b0f00e884339\") " pod="openshift-insights/insights-runtime-extractor-xnpz8" Jul 24 03:21:23.257926 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:23.257905 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-tsp28" Jul 24 03:21:23.380087 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:23.380023 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-tsp28"] Jul 24 03:21:23.382413 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:21:23.382389 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12956bc1_3664_46a8_97e6_be81968535d9.slice/crio-38d00f08c3abadad3aa40b5e6e919df823c1202ed2427a4e21ed29912309d11a WatchSource:0}: Error finding container 38d00f08c3abadad3aa40b5e6e919df823c1202ed2427a4e21ed29912309d11a: Status 404 returned error can't find the container with id 38d00f08c3abadad3aa40b5e6e919df823c1202ed2427a4e21ed29912309d11a Jul 24 03:21:23.662962 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:23.662936 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:23.717104 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:23.717073 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9373ab5f-eb4b-495c-9f7f-b0f00e884339-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xnpz8\" (UID: \"9373ab5f-eb4b-495c-9f7f-b0f00e884339\") " pod="openshift-insights/insights-runtime-extractor-xnpz8" Jul 24 03:21:23.717205 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:23.717181 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9373ab5f-eb4b-495c-9f7f-b0f00e884339-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xnpz8\" (UID: \"9373ab5f-eb4b-495c-9f7f-b0f00e884339\") " pod="openshift-insights/insights-runtime-extractor-xnpz8" Jul 24 03:21:23.717246 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:21:23.717206 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 03:21:23.717514 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:21:23.717447 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9373ab5f-eb4b-495c-9f7f-b0f00e884339-kube-rbac-proxy-cm podName:9373ab5f-eb4b-495c-9f7f-b0f00e884339 nodeName:}" failed. No retries permitted until 2026-07-24 03:21:24.717362323 +0000 UTC m=+234.156781761 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9373ab5f-eb4b-495c-9f7f-b0f00e884339-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xnpz8" (UID: "9373ab5f-eb4b-495c-9f7f-b0f00e884339") : configmap "kube-rbac-proxy" not found Jul 24 03:21:23.717677 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:21:23.717621 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 03:21:23.717737 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:21:23.717692 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9373ab5f-eb4b-495c-9f7f-b0f00e884339-insights-runtime-extractor-tls podName:9373ab5f-eb4b-495c-9f7f-b0f00e884339 nodeName:}" failed. No retries permitted until 2026-07-24 03:21:24.717674102 +0000 UTC m=+234.157093553 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9373ab5f-eb4b-495c-9f7f-b0f00e884339-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xnpz8" (UID: "9373ab5f-eb4b-495c-9f7f-b0f00e884339") : secret "insights-runtime-extractor-tls" not found Jul 24 03:21:23.763535 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:23.763499 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:23.865151 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:23.865117 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-tsp28" event={"ID":"12956bc1-3664-46a8-97e6-be81968535d9","Type":"ContainerStarted","Data":"38d00f08c3abadad3aa40b5e6e919df823c1202ed2427a4e21ed29912309d11a"} Jul 24 03:21:24.063120 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:24.063024 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:24.117640 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:24.117600 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:24.306184 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:24.306147 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:24.406587 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:24.406554 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:24.727708 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:24.727624 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9373ab5f-eb4b-495c-9f7f-b0f00e884339-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xnpz8\" (UID: \"9373ab5f-eb4b-495c-9f7f-b0f00e884339\") " pod="openshift-insights/insights-runtime-extractor-xnpz8" Jul 24 03:21:24.727869 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:24.727730 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9373ab5f-eb4b-495c-9f7f-b0f00e884339-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xnpz8\" (UID: \"9373ab5f-eb4b-495c-9f7f-b0f00e884339\") " pod="openshift-insights/insights-runtime-extractor-xnpz8" Jul 24 03:21:24.727869 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:21:24.727758 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 03:21:24.727869 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:21:24.727820 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9373ab5f-eb4b-495c-9f7f-b0f00e884339-kube-rbac-proxy-cm podName:9373ab5f-eb4b-495c-9f7f-b0f00e884339 nodeName:}" failed. No retries permitted until 2026-07-24 03:21:26.727804232 +0000 UTC m=+236.167223670 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9373ab5f-eb4b-495c-9f7f-b0f00e884339-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xnpz8" (UID: "9373ab5f-eb4b-495c-9f7f-b0f00e884339") : configmap "kube-rbac-proxy" not found Jul 24 03:21:24.727869 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:21:24.727837 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 03:21:24.728079 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:21:24.727894 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9373ab5f-eb4b-495c-9f7f-b0f00e884339-insights-runtime-extractor-tls podName:9373ab5f-eb4b-495c-9f7f-b0f00e884339 nodeName:}" failed. No retries permitted until 2026-07-24 03:21:26.727878396 +0000 UTC m=+236.167297851 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9373ab5f-eb4b-495c-9f7f-b0f00e884339-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xnpz8" (UID: "9373ab5f-eb4b-495c-9f7f-b0f00e884339") : secret "insights-runtime-extractor-tls" not found Jul 24 03:21:24.869695 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:24.869660 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-tsp28" event={"ID":"12956bc1-3664-46a8-97e6-be81968535d9","Type":"ContainerStarted","Data":"5f4d716fdb7da15d0d0498152ada902e2400d7c99d9f2ff3357bd2e265c0d00e"} Jul 24 03:21:24.869878 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:24.869726 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-tsp28" event={"ID":"12956bc1-3664-46a8-97e6-be81968535d9","Type":"ContainerStarted","Data":"a696ca464aa82c7b160a4bb4637616ddba6b6f1c6393d4a4acb61f0f991fa9d0"} Jul 24 03:21:24.883545 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:24.883502 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-tsp28" podStartSLOduration=1.761257796 podStartE2EDuration="2.883487577s" totalImagesPullingTime="1.122229781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:21:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:21:24.882839369 +0000 UTC m=+234.322258830" watchObservedRunningTime="2026-07-24 03:21:24.883487577 +0000 UTC m=+234.322907028" Jul 24 03:21:24.958211 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:24.958179 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:25.013043 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:25.012957 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:25.899055 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:25.899000 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-c25xr"] Jul 24 03:21:25.902127 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:25.902107 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-c25xr" Jul 24 03:21:25.904328 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:25.904308 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-lfzd2\"" Jul 24 03:21:25.904441 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:25.904419 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 24 03:21:25.905071 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:25.905054 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 24 03:21:25.905155 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:25.905060 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 24 03:21:25.905198 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:25.905061 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 24 03:21:25.910535 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:25.910509 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-c25xr"] Jul 24 03:21:25.933727 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:25.933705 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:25.959372 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:25.959350 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zgsks_a5b9b325-d029-42ca-8dfb-c75c2dd0c726/dns-node-resolver/0.log" Jul 24 03:21:26.037554 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:26.037522 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdl9r\" (UniqueName: \"kubernetes.io/projected/ddbe681b-dade-4cda-a056-0fa66137837b-kube-api-access-tdl9r\") pod \"service-ca-658d7b7c78-c25xr\" (UID: \"ddbe681b-dade-4cda-a056-0fa66137837b\") " pod="openshift-service-ca/service-ca-658d7b7c78-c25xr" Jul 24 03:21:26.037744 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:26.037605 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ddbe681b-dade-4cda-a056-0fa66137837b-signing-key\") pod \"service-ca-658d7b7c78-c25xr\" (UID: \"ddbe681b-dade-4cda-a056-0fa66137837b\") " pod="openshift-service-ca/service-ca-658d7b7c78-c25xr" Jul 24 03:21:26.037744 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:26.037632 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ddbe681b-dade-4cda-a056-0fa66137837b-signing-cabundle\") pod \"service-ca-658d7b7c78-c25xr\" (UID: \"ddbe681b-dade-4cda-a056-0fa66137837b\") " pod="openshift-service-ca/service-ca-658d7b7c78-c25xr" Jul 24 03:21:26.138218 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:26.138188 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ddbe681b-dade-4cda-a056-0fa66137837b-signing-key\") pod \"service-ca-658d7b7c78-c25xr\" (UID: \"ddbe681b-dade-4cda-a056-0fa66137837b\") " pod="openshift-service-ca/service-ca-658d7b7c78-c25xr" Jul 24 03:21:26.138218 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:26.138223 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ddbe681b-dade-4cda-a056-0fa66137837b-signing-cabundle\") pod \"service-ca-658d7b7c78-c25xr\" (UID: \"ddbe681b-dade-4cda-a056-0fa66137837b\") " pod="openshift-service-ca/service-ca-658d7b7c78-c25xr" Jul 24 03:21:26.138422 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:26.138270 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdl9r\" (UniqueName: \"kubernetes.io/projected/ddbe681b-dade-4cda-a056-0fa66137837b-kube-api-access-tdl9r\") pod \"service-ca-658d7b7c78-c25xr\" (UID: \"ddbe681b-dade-4cda-a056-0fa66137837b\") " pod="openshift-service-ca/service-ca-658d7b7c78-c25xr" Jul 24 03:21:26.138845 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:26.138821 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ddbe681b-dade-4cda-a056-0fa66137837b-signing-cabundle\") pod \"service-ca-658d7b7c78-c25xr\" (UID: \"ddbe681b-dade-4cda-a056-0fa66137837b\") " pod="openshift-service-ca/service-ca-658d7b7c78-c25xr" Jul 24 03:21:26.140776 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:26.140750 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ddbe681b-dade-4cda-a056-0fa66137837b-signing-key\") pod \"service-ca-658d7b7c78-c25xr\" (UID: \"ddbe681b-dade-4cda-a056-0fa66137837b\") " pod="openshift-service-ca/service-ca-658d7b7c78-c25xr" Jul 24 03:21:26.146074 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:26.146053 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdl9r\" (UniqueName: \"kubernetes.io/projected/ddbe681b-dade-4cda-a056-0fa66137837b-kube-api-access-tdl9r\") pod \"service-ca-658d7b7c78-c25xr\" (UID: \"ddbe681b-dade-4cda-a056-0fa66137837b\") " pod="openshift-service-ca/service-ca-658d7b7c78-c25xr" Jul 24 03:21:26.212002 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:26.211948 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-c25xr" Jul 24 03:21:26.333478 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:26.333428 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-c25xr"] Jul 24 03:21:26.336180 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:21:26.336156 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddbe681b_dade_4cda_a056_0fa66137837b.slice/crio-8e3bdde68a9f1072d62049494f7867e3ff0fba99cffcb14233b722208638abe5 WatchSource:0}: Error finding container 8e3bdde68a9f1072d62049494f7867e3ff0fba99cffcb14233b722208638abe5: Status 404 returned error can't find the container with id 8e3bdde68a9f1072d62049494f7867e3ff0fba99cffcb14233b722208638abe5 Jul 24 03:21:26.742568 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:26.742533 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9373ab5f-eb4b-495c-9f7f-b0f00e884339-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xnpz8\" (UID: \"9373ab5f-eb4b-495c-9f7f-b0f00e884339\") " pod="openshift-insights/insights-runtime-extractor-xnpz8" Jul 24 03:21:26.742747 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:26.742598 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9373ab5f-eb4b-495c-9f7f-b0f00e884339-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xnpz8\" (UID: \"9373ab5f-eb4b-495c-9f7f-b0f00e884339\") " pod="openshift-insights/insights-runtime-extractor-xnpz8" Jul 24 03:21:26.742747 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:21:26.742656 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 03:21:26.742747 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:21:26.742723 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 03:21:26.742847 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:21:26.742726 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9373ab5f-eb4b-495c-9f7f-b0f00e884339-kube-rbac-proxy-cm podName:9373ab5f-eb4b-495c-9f7f-b0f00e884339 nodeName:}" failed. No retries permitted until 2026-07-24 03:21:30.742707494 +0000 UTC m=+240.182126933 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9373ab5f-eb4b-495c-9f7f-b0f00e884339-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xnpz8" (UID: "9373ab5f-eb4b-495c-9f7f-b0f00e884339") : configmap "kube-rbac-proxy" not found Jul 24 03:21:26.742847 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:21:26.742783 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9373ab5f-eb4b-495c-9f7f-b0f00e884339-insights-runtime-extractor-tls podName:9373ab5f-eb4b-495c-9f7f-b0f00e884339 nodeName:}" failed. No retries permitted until 2026-07-24 03:21:30.742767955 +0000 UTC m=+240.182187395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9373ab5f-eb4b-495c-9f7f-b0f00e884339-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xnpz8" (UID: "9373ab5f-eb4b-495c-9f7f-b0f00e884339") : secret "insights-runtime-extractor-tls" not found Jul 24 03:21:26.764039 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:26.764019 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8brgv_daacf9bc-9977-408a-b1b4-cd361f3cf528/node-ca/0.log" Jul 24 03:21:26.765726 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:26.765695 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:26.879113 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:26.879073 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-c25xr" event={"ID":"ddbe681b-dade-4cda-a056-0fa66137837b","Type":"ContainerStarted","Data":"90433449cec4b6f96f906e8aae18cf5c871c7f6d33bdebe7dea40f7d734a5e5c"} Jul 24 03:21:26.879289 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:26.879138 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-c25xr" event={"ID":"ddbe681b-dade-4cda-a056-0fa66137837b","Type":"ContainerStarted","Data":"8e3bdde68a9f1072d62049494f7867e3ff0fba99cffcb14233b722208638abe5"} Jul 24 03:21:26.899435 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:26.899388 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-c25xr" podStartSLOduration=1.8993697250000001 podStartE2EDuration="1.899369725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:21:26.898135999 +0000 UTC m=+236.337555459" watchObservedRunningTime="2026-07-24 03:21:26.899369725 +0000 UTC m=+236.338789185" Jul 24 03:21:27.063995 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:27.063910 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:27.270652 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:27.270599 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:27.370845 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:27.370808 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:27.471450 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:27.471421 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:27.560846 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:27.560807 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-75ljs_e63f0954-7da9-4fdc-a43d-d1cc3323aaa1/serve-healthcheck-canary/0.log" Jul 24 03:21:27.587409 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:27.587376 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:27.934832 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:27.934801 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:27.960159 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:27.960120 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-tsp28_12956bc1-3664-46a8-97e6-be81968535d9/migrator/0.log" Jul 24 03:21:28.153184 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:28.153146 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:28.160393 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:28.160369 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-tsp28_12956bc1-3664-46a8-97e6-be81968535d9/graceful-termination/0.log" Jul 24 03:21:28.949111 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:28.949079 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:29.357505 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:29.357456 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:29.547984 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:29.547947 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:29.747730 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:29.747651 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:30.152389 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:30.152347 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:30.548627 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:30.548540 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:30.775588 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:30.775547 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9373ab5f-eb4b-495c-9f7f-b0f00e884339-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xnpz8\" (UID: \"9373ab5f-eb4b-495c-9f7f-b0f00e884339\") " pod="openshift-insights/insights-runtime-extractor-xnpz8" Jul 24 03:21:30.775752 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:30.775608 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9373ab5f-eb4b-495c-9f7f-b0f00e884339-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xnpz8\" (UID: \"9373ab5f-eb4b-495c-9f7f-b0f00e884339\") " pod="openshift-insights/insights-runtime-extractor-xnpz8" Jul 24 03:21:30.775752 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:21:30.775663 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 03:21:30.775752 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:21:30.775720 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 03:21:30.775752 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:21:30.775731 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9373ab5f-eb4b-495c-9f7f-b0f00e884339-kube-rbac-proxy-cm podName:9373ab5f-eb4b-495c-9f7f-b0f00e884339 nodeName:}" failed. No retries permitted until 2026-07-24 03:21:38.775714578 +0000 UTC m=+248.215134018 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9373ab5f-eb4b-495c-9f7f-b0f00e884339-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xnpz8" (UID: "9373ab5f-eb4b-495c-9f7f-b0f00e884339") : configmap "kube-rbac-proxy" not found Jul 24 03:21:30.775885 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:21:30.775760 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9373ab5f-eb4b-495c-9f7f-b0f00e884339-insights-runtime-extractor-tls podName:9373ab5f-eb4b-495c-9f7f-b0f00e884339 nodeName:}" failed. No retries permitted until 2026-07-24 03:21:38.775748594 +0000 UTC m=+248.215168032 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9373ab5f-eb4b-495c-9f7f-b0f00e884339-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xnpz8" (UID: "9373ab5f-eb4b-495c-9f7f-b0f00e884339") : secret "insights-runtime-extractor-tls" not found Jul 24 03:21:30.949798 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:30.949766 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:31.050478 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:31.050436 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:31.154666 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:31.154635 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:31.751691 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:31.751651 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:32.163682 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:32.163650 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:32.554268 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:32.554180 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:32.759749 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:32.759711 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:32.904039 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:32.904002 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:33.151420 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:33.151380 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:33.559186 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:33.559149 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:33.635762 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:33.635728 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:34.154769 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:34.154734 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:34.353005 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:34.352970 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:34.949891 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:34.949861 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:35.548310 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:35.548278 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:35.753383 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:35.753354 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:35.951400 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:35.951362 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:36.088438 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:36.088404 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:36.350673 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:36.350583 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:36.947332 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:36.947302 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:37.377721 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:37.377693 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:37.478274 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:37.478240 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:37.579149 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:37.579118 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:37.633823 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:37.633743 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:37.745582 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:37.745550 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:38.149731 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:38.149697 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:38.841480 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:38.841419 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9373ab5f-eb4b-495c-9f7f-b0f00e884339-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xnpz8\" (UID: \"9373ab5f-eb4b-495c-9f7f-b0f00e884339\") " pod="openshift-insights/insights-runtime-extractor-xnpz8" Jul 24 03:21:38.841944 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:38.841514 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9373ab5f-eb4b-495c-9f7f-b0f00e884339-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xnpz8\" (UID: \"9373ab5f-eb4b-495c-9f7f-b0f00e884339\") " pod="openshift-insights/insights-runtime-extractor-xnpz8" Jul 24 03:21:38.842078 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:38.842057 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9373ab5f-eb4b-495c-9f7f-b0f00e884339-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xnpz8\" (UID: \"9373ab5f-eb4b-495c-9f7f-b0f00e884339\") " pod="openshift-insights/insights-runtime-extractor-xnpz8" Jul 24 03:21:38.843815 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:38.843795 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9373ab5f-eb4b-495c-9f7f-b0f00e884339-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xnpz8\" (UID: \"9373ab5f-eb4b-495c-9f7f-b0f00e884339\") " pod="openshift-insights/insights-runtime-extractor-xnpz8" Jul 24 03:21:38.878579 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:38.878554 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fs6xn\"" Jul 24 03:21:38.886270 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:38.886254 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xnpz8" Jul 24 03:21:39.012271 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:39.012250 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xnpz8"] Jul 24 03:21:39.014656 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:21:39.014628 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9373ab5f_eb4b_495c_9f7f_b0f00e884339.slice/crio-edc6d464e9552c95c5614a88e53c2d2a67342cc9ee781c2b2fdb2a8843d99db6 WatchSource:0}: Error finding container edc6d464e9552c95c5614a88e53c2d2a67342cc9ee781c2b2fdb2a8843d99db6: Status 404 returned error can't find the container with id edc6d464e9552c95c5614a88e53c2d2a67342cc9ee781c2b2fdb2a8843d99db6 Jul 24 03:21:39.926126 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:39.926084 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xnpz8" event={"ID":"9373ab5f-eb4b-495c-9f7f-b0f00e884339","Type":"ContainerStarted","Data":"edc6d464e9552c95c5614a88e53c2d2a67342cc9ee781c2b2fdb2a8843d99db6"} Jul 24 03:21:42.774145 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:42.774094 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs\") pod \"network-metrics-daemon-8xwcn\" (UID: \"8fdc01f1-f7a1-4395-bae7-be0c9d76b76c\") " pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:21:42.777013 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:42.776985 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fdc01f1-f7a1-4395-bae7-be0c9d76b76c-metrics-certs\") pod \"network-metrics-daemon-8xwcn\" (UID: \"8fdc01f1-f7a1-4395-bae7-be0c9d76b76c\") " pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:21:42.930005 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:42.929973 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-frl8x\"" Jul 24 03:21:42.937643 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:42.937613 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xwcn" Jul 24 03:21:44.005916 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:44.005886 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8xwcn"] Jul 24 03:21:44.009668 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:21:44.009637 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fdc01f1_f7a1_4395_bae7_be0c9d76b76c.slice/crio-7b0a82e568ff8549a141e812a50a6a31c8e0092526249add167800008893ff3d WatchSource:0}: Error finding container 7b0a82e568ff8549a141e812a50a6a31c8e0092526249add167800008893ff3d: Status 404 returned error can't find the container with id 7b0a82e568ff8549a141e812a50a6a31c8e0092526249add167800008893ff3d Jul 24 03:21:44.951015 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:44.950906 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xnpz8" event={"ID":"9373ab5f-eb4b-495c-9f7f-b0f00e884339","Type":"ContainerStarted","Data":"3e3dd928ad48d511557a90c2acdddbc2a3657d7fff290e0c2f44a15a90950a1c"} Jul 24 03:21:44.952387 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:44.952362 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8xwcn" event={"ID":"8fdc01f1-f7a1-4395-bae7-be0c9d76b76c","Type":"ContainerStarted","Data":"7b0a82e568ff8549a141e812a50a6a31c8e0092526249add167800008893ff3d"} Jul 24 03:21:45.082109 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.081750 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-lh52r"] Jul 24 03:21:45.086227 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.086194 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-lh52r" Jul 24 03:21:45.089339 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.089141 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-4mrsz\"" Jul 24 03:21:45.090160 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.090111 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 24 03:21:45.092791 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.092299 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ab8bad1f-dacb-4db0-af0b-3efb426a5405-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-lh52r\" (UID: \"ab8bad1f-dacb-4db0-af0b-3efb426a5405\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-lh52r" Jul 24 03:21:45.096817 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.096788 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-lh52r"] Jul 24 03:21:45.180219 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.180193 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-77678456c6-kmzrp"] Jul 24 03:21:45.183113 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.183091 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77678456c6-kmzrp" Jul 24 03:21:45.186363 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.186209 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 24 03:21:45.186363 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.186232 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bwrkh\"" Jul 24 03:21:45.186767 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.186745 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 24 03:21:45.186874 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.186815 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 24 03:21:45.191260 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.191164 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 24 03:21:45.193505 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.193476 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/003896e6-6475-4611-92c7-9beea3b43ae9-registry-tls\") pod \"image-registry-77678456c6-kmzrp\" (UID: \"003896e6-6475-4611-92c7-9beea3b43ae9\") " pod="openshift-image-registry/image-registry-77678456c6-kmzrp" Jul 24 03:21:45.193605 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.193523 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ab8bad1f-dacb-4db0-af0b-3efb426a5405-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-lh52r\" (UID: \"ab8bad1f-dacb-4db0-af0b-3efb426a5405\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-lh52r" Jul 24 03:21:45.193605 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.193550 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/003896e6-6475-4611-92c7-9beea3b43ae9-registry-certificates\") pod \"image-registry-77678456c6-kmzrp\" (UID: \"003896e6-6475-4611-92c7-9beea3b43ae9\") " pod="openshift-image-registry/image-registry-77678456c6-kmzrp" Jul 24 03:21:45.193605 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.193575 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/003896e6-6475-4611-92c7-9beea3b43ae9-installation-pull-secrets\") pod \"image-registry-77678456c6-kmzrp\" (UID: \"003896e6-6475-4611-92c7-9beea3b43ae9\") " pod="openshift-image-registry/image-registry-77678456c6-kmzrp" Jul 24 03:21:45.193766 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.193622 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/003896e6-6475-4611-92c7-9beea3b43ae9-image-registry-private-configuration\") pod \"image-registry-77678456c6-kmzrp\" (UID: \"003896e6-6475-4611-92c7-9beea3b43ae9\") " pod="openshift-image-registry/image-registry-77678456c6-kmzrp" Jul 24 03:21:45.193766 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.193654 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqkdh\" (UniqueName: \"kubernetes.io/projected/003896e6-6475-4611-92c7-9beea3b43ae9-kube-api-access-rqkdh\") pod \"image-registry-77678456c6-kmzrp\" (UID: \"003896e6-6475-4611-92c7-9beea3b43ae9\") " pod="openshift-image-registry/image-registry-77678456c6-kmzrp" Jul 24 03:21:45.193766 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.193681 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/003896e6-6475-4611-92c7-9beea3b43ae9-ca-trust-extracted\") pod \"image-registry-77678456c6-kmzrp\" (UID: \"003896e6-6475-4611-92c7-9beea3b43ae9\") " pod="openshift-image-registry/image-registry-77678456c6-kmzrp" Jul 24 03:21:45.193766 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.193709 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/003896e6-6475-4611-92c7-9beea3b43ae9-trusted-ca\") pod \"image-registry-77678456c6-kmzrp\" (UID: \"003896e6-6475-4611-92c7-9beea3b43ae9\") " pod="openshift-image-registry/image-registry-77678456c6-kmzrp" Jul 24 03:21:45.193766 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.193754 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/003896e6-6475-4611-92c7-9beea3b43ae9-bound-sa-token\") pod \"image-registry-77678456c6-kmzrp\" (UID: \"003896e6-6475-4611-92c7-9beea3b43ae9\") " pod="openshift-image-registry/image-registry-77678456c6-kmzrp" Jul 24 03:21:45.197215 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.197167 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ab8bad1f-dacb-4db0-af0b-3efb426a5405-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-lh52r\" (UID: \"ab8bad1f-dacb-4db0-af0b-3efb426a5405\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-lh52r" Jul 24 03:21:45.200516 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.200436 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77678456c6-kmzrp"] Jul 24 03:21:45.235277 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.235217 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:45.294039 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.294009 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/003896e6-6475-4611-92c7-9beea3b43ae9-registry-tls\") pod \"image-registry-77678456c6-kmzrp\" (UID: \"003896e6-6475-4611-92c7-9beea3b43ae9\") " pod="openshift-image-registry/image-registry-77678456c6-kmzrp" Jul 24 03:21:45.294146 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.294055 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/003896e6-6475-4611-92c7-9beea3b43ae9-registry-certificates\") pod \"image-registry-77678456c6-kmzrp\" (UID: \"003896e6-6475-4611-92c7-9beea3b43ae9\") " pod="openshift-image-registry/image-registry-77678456c6-kmzrp" Jul 24 03:21:45.294146 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.294082 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/003896e6-6475-4611-92c7-9beea3b43ae9-installation-pull-secrets\") pod \"image-registry-77678456c6-kmzrp\" (UID: \"003896e6-6475-4611-92c7-9beea3b43ae9\") " pod="openshift-image-registry/image-registry-77678456c6-kmzrp" Jul 24 03:21:45.294146 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.294124 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/003896e6-6475-4611-92c7-9beea3b43ae9-image-registry-private-configuration\") pod \"image-registry-77678456c6-kmzrp\" (UID: \"003896e6-6475-4611-92c7-9beea3b43ae9\") " pod="openshift-image-registry/image-registry-77678456c6-kmzrp" Jul 24 03:21:45.294282 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.294153 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqkdh\" (UniqueName: \"kubernetes.io/projected/003896e6-6475-4611-92c7-9beea3b43ae9-kube-api-access-rqkdh\") pod \"image-registry-77678456c6-kmzrp\" (UID: \"003896e6-6475-4611-92c7-9beea3b43ae9\") " pod="openshift-image-registry/image-registry-77678456c6-kmzrp" Jul 24 03:21:45.294282 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.294190 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/003896e6-6475-4611-92c7-9beea3b43ae9-ca-trust-extracted\") pod \"image-registry-77678456c6-kmzrp\" (UID: \"003896e6-6475-4611-92c7-9beea3b43ae9\") " pod="openshift-image-registry/image-registry-77678456c6-kmzrp" Jul 24 03:21:45.294282 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.294229 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/003896e6-6475-4611-92c7-9beea3b43ae9-trusted-ca\") pod \"image-registry-77678456c6-kmzrp\" (UID: \"003896e6-6475-4611-92c7-9beea3b43ae9\") " pod="openshift-image-registry/image-registry-77678456c6-kmzrp" Jul 24 03:21:45.294282 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.294271 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/003896e6-6475-4611-92c7-9beea3b43ae9-bound-sa-token\") pod \"image-registry-77678456c6-kmzrp\" (UID: \"003896e6-6475-4611-92c7-9beea3b43ae9\") " pod="openshift-image-registry/image-registry-77678456c6-kmzrp" Jul 24 03:21:45.295835 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.295006 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/003896e6-6475-4611-92c7-9beea3b43ae9-ca-trust-extracted\") pod \"image-registry-77678456c6-kmzrp\" (UID: \"003896e6-6475-4611-92c7-9beea3b43ae9\") " pod="openshift-image-registry/image-registry-77678456c6-kmzrp" Jul 24 03:21:45.295835 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.295789 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/003896e6-6475-4611-92c7-9beea3b43ae9-registry-certificates\") pod \"image-registry-77678456c6-kmzrp\" (UID: \"003896e6-6475-4611-92c7-9beea3b43ae9\") " pod="openshift-image-registry/image-registry-77678456c6-kmzrp" Jul 24 03:21:45.296220 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.296076 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/003896e6-6475-4611-92c7-9beea3b43ae9-trusted-ca\") pod \"image-registry-77678456c6-kmzrp\" (UID: \"003896e6-6475-4611-92c7-9beea3b43ae9\") " pod="openshift-image-registry/image-registry-77678456c6-kmzrp" Jul 24 03:21:45.297693 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.297650 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/003896e6-6475-4611-92c7-9beea3b43ae9-installation-pull-secrets\") pod \"image-registry-77678456c6-kmzrp\" (UID: \"003896e6-6475-4611-92c7-9beea3b43ae9\") " pod="openshift-image-registry/image-registry-77678456c6-kmzrp" Jul 24 03:21:45.297871 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.297826 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/003896e6-6475-4611-92c7-9beea3b43ae9-image-registry-private-configuration\") pod \"image-registry-77678456c6-kmzrp\" (UID: \"003896e6-6475-4611-92c7-9beea3b43ae9\") " pod="openshift-image-registry/image-registry-77678456c6-kmzrp" Jul 24 03:21:45.298208 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.298190 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/003896e6-6475-4611-92c7-9beea3b43ae9-registry-tls\") pod \"image-registry-77678456c6-kmzrp\" (UID: \"003896e6-6475-4611-92c7-9beea3b43ae9\") " pod="openshift-image-registry/image-registry-77678456c6-kmzrp" Jul 24 03:21:45.311080 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.311043 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/003896e6-6475-4611-92c7-9beea3b43ae9-bound-sa-token\") pod \"image-registry-77678456c6-kmzrp\" (UID: \"003896e6-6475-4611-92c7-9beea3b43ae9\") " pod="openshift-image-registry/image-registry-77678456c6-kmzrp" Jul 24 03:21:45.313598 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.313551 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqkdh\" (UniqueName: \"kubernetes.io/projected/003896e6-6475-4611-92c7-9beea3b43ae9-kube-api-access-rqkdh\") pod \"image-registry-77678456c6-kmzrp\" (UID: \"003896e6-6475-4611-92c7-9beea3b43ae9\") " pod="openshift-image-registry/image-registry-77678456c6-kmzrp" Jul 24 03:21:45.370569 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.370539 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:45.399131 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.399096 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-lh52r" Jul 24 03:21:45.459133 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.459068 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:45.513885 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.513788 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77678456c6-kmzrp" Jul 24 03:21:45.553406 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.553376 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-lh52r"] Jul 24 03:21:45.557168 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.557124 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:45.559294 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:21:45.559267 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab8bad1f_dacb_4db0_af0b_3efb426a5405.slice/crio-2c44245f6ff5682981218b0f537f57e04b364b2cc3e384d8129f785e3d43ec9b WatchSource:0}: Error finding container 2c44245f6ff5682981218b0f537f57e04b364b2cc3e384d8129f785e3d43ec9b: Status 404 returned error can't find the container with id 2c44245f6ff5682981218b0f537f57e04b364b2cc3e384d8129f785e3d43ec9b Jul 24 03:21:45.672449 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.672409 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77678456c6-kmzrp"] Jul 24 03:21:45.753313 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.753226 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:45.934631 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.934583 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:45.958559 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.958522 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8xwcn" event={"ID":"8fdc01f1-f7a1-4395-bae7-be0c9d76b76c","Type":"ContainerStarted","Data":"a6d1a44d3351171f327ed8e1b9ec1f9152045de19e8a57b32aca7685004e8129"} Jul 24 03:21:45.958743 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.958585 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8xwcn" event={"ID":"8fdc01f1-f7a1-4395-bae7-be0c9d76b76c","Type":"ContainerStarted","Data":"40fb3607f354f6958ad31d9531685bafdd3b67a2cca27a2b435ca53da4afd1f1"} Jul 24 03:21:45.959826 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:45.959799 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-lh52r" event={"ID":"ab8bad1f-dacb-4db0-af0b-3efb426a5405","Type":"ContainerStarted","Data":"2c44245f6ff5682981218b0f537f57e04b364b2cc3e384d8129f785e3d43ec9b"} Jul 24 03:21:46.035834 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:46.035711 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:46.135758 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:46.135721 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:46.191406 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:46.191373 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:46.334012 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:46.333914 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:46.435176 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:46.435141 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:46.852595 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:21:46.852558 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003896e6_6475_4611_92c7_9beea3b43ae9.slice/crio-af7706486942ca5cafdb246043e58c4ce58f0a2425124d0f3cf95d676ffa60de WatchSource:0}: Error finding container af7706486942ca5cafdb246043e58c4ce58f0a2425124d0f3cf95d676ffa60de: Status 404 returned error can't find the container with id af7706486942ca5cafdb246043e58c4ce58f0a2425124d0f3cf95d676ffa60de Jul 24 03:21:46.965270 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:46.965232 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77678456c6-kmzrp" event={"ID":"003896e6-6475-4611-92c7-9beea3b43ae9","Type":"ContainerStarted","Data":"af7706486942ca5cafdb246043e58c4ce58f0a2425124d0f3cf95d676ffa60de"} Jul 24 03:21:47.214995 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:47.214952 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:47.417143 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:47.417056 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:47.971115 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:47.970676 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77678456c6-kmzrp" event={"ID":"003896e6-6475-4611-92c7-9beea3b43ae9","Type":"ContainerStarted","Data":"0166b8e5d4e52dd1478c10d85cc000fe2b16dc7530a87490af6070c1f0ad7ba5"} Jul 24 03:21:47.971115 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:47.970828 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-77678456c6-kmzrp" Jul 24 03:21:47.989917 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:47.989871 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8xwcn" podStartSLOduration=255.828243382 podStartE2EDuration="4m16.989854262s" totalImagesPullingTime="1.16161088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:21:45.973395908 +0000 UTC m=+255.412815369" watchObservedRunningTime="2026-07-24 03:21:47.989854262 +0000 UTC m=+257.429273742" Jul 24 03:21:47.990219 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:47.990163 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-77678456c6-kmzrp" podStartSLOduration=2.990157456 podStartE2EDuration="2.990157456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:21:47.988077633 +0000 UTC m=+257.427497095" watchObservedRunningTime="2026-07-24 03:21:47.990157456 +0000 UTC m=+257.429576910" Jul 24 03:21:48.034519 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:48.034485 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:48.824471 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:48.824425 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:48.977253 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:48.977205 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xnpz8" event={"ID":"9373ab5f-eb4b-495c-9f7f-b0f00e884339","Type":"ContainerStarted","Data":"86d9379b51fd1bdd8658d6eb46e6d324f29e47bc0e3a15479c32243c6fdfe8eb"} Jul 24 03:21:48.978873 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:48.978842 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-lh52r" event={"ID":"ab8bad1f-dacb-4db0-af0b-3efb426a5405","Type":"ContainerStarted","Data":"2923e03e41730d0d4cab7d475063bbc88aef9f82fa1909afe2b60750e62254a3"} Jul 24 03:21:48.992685 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:48.992614 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-lh52r" podStartSLOduration=1.618673303 podStartE2EDuration="3.99259639s" totalImagesPullingTime="2.373923087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:21:48.992069368 +0000 UTC m=+258.431488830" watchObservedRunningTime="2026-07-24 03:21:48.99259639 +0000 UTC m=+258.432015849" Jul 24 03:21:49.982345 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:49.982312 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-lh52r" Jul 24 03:21:49.988121 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:49.988096 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-lh52r" Jul 24 03:21:50.602503 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:50.602445 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:50.672926 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:50.672876 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:50.997994 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:50.997957 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:51.990377 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:51.990292 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xnpz8" event={"ID":"9373ab5f-eb4b-495c-9f7f-b0f00e884339","Type":"ContainerStarted","Data":"fa51574babb6a19109fbb4a92c7b36b88a77fd42e5467ed71315cca1c2f3f5b9"} Jul 24 03:21:52.007017 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:52.006951 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xnpz8" podStartSLOduration=17.787240382 podStartE2EDuration="30.006930766s" totalImagesPullingTime="12.219690384s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 03:21:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:21:52.005969993 +0000 UTC m=+261.445389452" watchObservedRunningTime="2026-07-24 03:21:52.006930766 +0000 UTC m=+261.446350227" Jul 24 03:21:53.226570 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:53.226531 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:53.326135 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:53.326095 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:53.441520 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:53.441488 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:55.731297 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:55.731223 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:55.835441 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:55.835405 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:55.954207 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:55.954171 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:55.982955 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:55.982884 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-82q8v"] Jul 24 03:21:55.987848 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:55.987828 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:55.990797 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:55.990776 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 24 03:21:55.990955 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:55.990936 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 24 03:21:55.991087 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:55.991071 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 24 03:21:55.991237 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:55.991015 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 24 03:21:55.992840 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:55.992784 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 24 03:21:55.992840 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:55.992807 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 24 03:21:55.992840 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:55.992787 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2wwgj\"" Jul 24 03:21:56.037514 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.037476 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:56.076786 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.076747 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttjbl\" (UniqueName: \"kubernetes.io/projected/31e8d580-562b-482b-9c7b-673cf95070ea-kube-api-access-ttjbl\") pod \"node-exporter-82q8v\" (UID: \"31e8d580-562b-482b-9c7b-673cf95070ea\") " pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.076939 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.076810 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31e8d580-562b-482b-9c7b-673cf95070ea-sys\") pod \"node-exporter-82q8v\" (UID: \"31e8d580-562b-482b-9c7b-673cf95070ea\") " pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.076939 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.076871 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/31e8d580-562b-482b-9c7b-673cf95070ea-node-exporter-textfile\") pod \"node-exporter-82q8v\" (UID: \"31e8d580-562b-482b-9c7b-673cf95070ea\") " pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.077018 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.076938 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/31e8d580-562b-482b-9c7b-673cf95070ea-node-exporter-accelerators-collector-config\") pod \"node-exporter-82q8v\" (UID: \"31e8d580-562b-482b-9c7b-673cf95070ea\") " pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.077018 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.076965 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/31e8d580-562b-482b-9c7b-673cf95070ea-node-exporter-tls\") pod \"node-exporter-82q8v\" (UID: \"31e8d580-562b-482b-9c7b-673cf95070ea\") " pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.077018 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.076983 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/31e8d580-562b-482b-9c7b-673cf95070ea-node-exporter-wtmp\") pod \"node-exporter-82q8v\" (UID: \"31e8d580-562b-482b-9c7b-673cf95070ea\") " pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.077018 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.077003 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31e8d580-562b-482b-9c7b-673cf95070ea-metrics-client-ca\") pod \"node-exporter-82q8v\" (UID: \"31e8d580-562b-482b-9c7b-673cf95070ea\") " pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.077153 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.077064 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/31e8d580-562b-482b-9c7b-673cf95070ea-root\") pod \"node-exporter-82q8v\" (UID: \"31e8d580-562b-482b-9c7b-673cf95070ea\") " pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.077153 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.077105 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31e8d580-562b-482b-9c7b-673cf95070ea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-82q8v\" (UID: \"31e8d580-562b-482b-9c7b-673cf95070ea\") " pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.119043 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.119010 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:56.178359 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.178319 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttjbl\" (UniqueName: \"kubernetes.io/projected/31e8d580-562b-482b-9c7b-673cf95070ea-kube-api-access-ttjbl\") pod \"node-exporter-82q8v\" (UID: \"31e8d580-562b-482b-9c7b-673cf95070ea\") " pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.178554 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.178380 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31e8d580-562b-482b-9c7b-673cf95070ea-sys\") pod \"node-exporter-82q8v\" (UID: \"31e8d580-562b-482b-9c7b-673cf95070ea\") " pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.178554 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.178441 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31e8d580-562b-482b-9c7b-673cf95070ea-sys\") pod \"node-exporter-82q8v\" (UID: \"31e8d580-562b-482b-9c7b-673cf95070ea\") " pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.178651 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.178560 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/31e8d580-562b-482b-9c7b-673cf95070ea-node-exporter-textfile\") pod \"node-exporter-82q8v\" (UID: \"31e8d580-562b-482b-9c7b-673cf95070ea\") " pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.178651 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.178612 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/31e8d580-562b-482b-9c7b-673cf95070ea-node-exporter-accelerators-collector-config\") pod \"node-exporter-82q8v\" (UID: \"31e8d580-562b-482b-9c7b-673cf95070ea\") " pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.178745 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.178649 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/31e8d580-562b-482b-9c7b-673cf95070ea-node-exporter-tls\") pod \"node-exporter-82q8v\" (UID: \"31e8d580-562b-482b-9c7b-673cf95070ea\") " pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.178745 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.178677 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/31e8d580-562b-482b-9c7b-673cf95070ea-node-exporter-wtmp\") pod \"node-exporter-82q8v\" (UID: \"31e8d580-562b-482b-9c7b-673cf95070ea\") " pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.178745 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.178700 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31e8d580-562b-482b-9c7b-673cf95070ea-metrics-client-ca\") pod \"node-exporter-82q8v\" (UID: \"31e8d580-562b-482b-9c7b-673cf95070ea\") " pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.178745 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.178733 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/31e8d580-562b-482b-9c7b-673cf95070ea-root\") pod \"node-exporter-82q8v\" (UID: \"31e8d580-562b-482b-9c7b-673cf95070ea\") " pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.178936 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.178757 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31e8d580-562b-482b-9c7b-673cf95070ea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-82q8v\" (UID: \"31e8d580-562b-482b-9c7b-673cf95070ea\") " pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.178936 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:21:56.178789 2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 24 03:21:56.178936 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:21:56.178848 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31e8d580-562b-482b-9c7b-673cf95070ea-node-exporter-tls podName:31e8d580-562b-482b-9c7b-673cf95070ea nodeName:}" failed. No retries permitted until 2026-07-24 03:21:56.678829199 +0000 UTC m=+266.118248642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/31e8d580-562b-482b-9c7b-673cf95070ea-node-exporter-tls") pod "node-exporter-82q8v" (UID: "31e8d580-562b-482b-9c7b-673cf95070ea") : secret "node-exporter-tls" not found Jul 24 03:21:56.178936 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.178895 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/31e8d580-562b-482b-9c7b-673cf95070ea-node-exporter-wtmp\") pod \"node-exporter-82q8v\" (UID: \"31e8d580-562b-482b-9c7b-673cf95070ea\") " pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.179133 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.178934 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/31e8d580-562b-482b-9c7b-673cf95070ea-node-exporter-textfile\") pod \"node-exporter-82q8v\" (UID: \"31e8d580-562b-482b-9c7b-673cf95070ea\") " pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.179133 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.179015 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/31e8d580-562b-482b-9c7b-673cf95070ea-root\") pod \"node-exporter-82q8v\" (UID: \"31e8d580-562b-482b-9c7b-673cf95070ea\") " pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.179490 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.179448 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31e8d580-562b-482b-9c7b-673cf95070ea-metrics-client-ca\") pod \"node-exporter-82q8v\" (UID: \"31e8d580-562b-482b-9c7b-673cf95070ea\") " pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.179611 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.179515 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/31e8d580-562b-482b-9c7b-673cf95070ea-node-exporter-accelerators-collector-config\") pod \"node-exporter-82q8v\" (UID: \"31e8d580-562b-482b-9c7b-673cf95070ea\") " pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.181640 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.181620 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31e8d580-562b-482b-9c7b-673cf95070ea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-82q8v\" (UID: \"31e8d580-562b-482b-9c7b-673cf95070ea\") " pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.188335 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.188292 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttjbl\" (UniqueName: \"kubernetes.io/projected/31e8d580-562b-482b-9c7b-673cf95070ea-kube-api-access-ttjbl\") pod \"node-exporter-82q8v\" (UID: \"31e8d580-562b-482b-9c7b-673cf95070ea\") " pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.238629 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.238540 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:56.338994 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.338950 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:56.444397 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.444360 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:56.498778 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.498690 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:56.682788 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.682749 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/31e8d580-562b-482b-9c7b-673cf95070ea-node-exporter-tls\") pod \"node-exporter-82q8v\" (UID: \"31e8d580-562b-482b-9c7b-673cf95070ea\") " pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.685208 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.685177 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/31e8d580-562b-482b-9c7b-673cf95070ea-node-exporter-tls\") pod \"node-exporter-82q8v\" (UID: \"31e8d580-562b-482b-9c7b-673cf95070ea\") " pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.900513 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.900449 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-82q8v" Jul 24 03:21:56.913048 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:21:56.913001 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e8d580_562b_482b_9c7b_673cf95070ea.slice/crio-8b063e55997a940ee654c85457cdda37b55411f50cdc3da8f607da4cfc0f32af WatchSource:0}: Error finding container 8b063e55997a940ee654c85457cdda37b55411f50cdc3da8f607da4cfc0f32af: Status 404 returned error can't find the container with id 8b063e55997a940ee654c85457cdda37b55411f50cdc3da8f607da4cfc0f32af Jul 24 03:21:56.967921 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:56.967882 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:57.009945 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:57.009910 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-82q8v" event={"ID":"31e8d580-562b-482b-9c7b-673cf95070ea","Type":"ContainerStarted","Data":"8b063e55997a940ee654c85457cdda37b55411f50cdc3da8f607da4cfc0f32af"} Jul 24 03:21:57.051189 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:57.051110 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:57.111527 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:57.111484 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:58.015145 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:58.015111 2579 generic.go:356] "Generic (PLEG): container finished" podID="31e8d580-562b-482b-9c7b-673cf95070ea" containerID="cd8472987c1e24cac4914e1fcbab2921dbe4b3e9cd6360ea7ce4efe389833c62" exitCode=0 Jul 24 03:21:58.015145 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:58.015148 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-82q8v" event={"ID":"31e8d580-562b-482b-9c7b-673cf95070ea","Type":"ContainerDied","Data":"cd8472987c1e24cac4914e1fcbab2921dbe4b3e9cd6360ea7ce4efe389833c62"} Jul 24 03:21:58.880798 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:58.880761 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:58.942439 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:58.942405 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:59.021164 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:59.021128 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-82q8v" event={"ID":"31e8d580-562b-482b-9c7b-673cf95070ea","Type":"ContainerStarted","Data":"dcaa38030aff75a1f2702befadee498508842be18e719feb28198a2cb03e6acd"} Jul 24 03:21:59.021585 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:59.021184 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-82q8v" event={"ID":"31e8d580-562b-482b-9c7b-673cf95070ea","Type":"ContainerStarted","Data":"110bd34044fc5663da0dba13d79bdb7d7213ac53ce22c0b54f258206a0d9a162"} Jul 24 03:21:59.040203 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:59.040141 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-82q8v" podStartSLOduration=3.047505357 podStartE2EDuration="4.040120886s" totalImagesPullingTime="923.391533ms" totalInitContainerRuntime="69.223996ms" isStatefulPod=true podCreationTimestamp="2026-07-24 03:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:21:59.039722848 +0000 UTC m=+268.479142308" watchObservedRunningTime="2026-07-24 03:21:59.040120886 +0000 UTC m=+268.479540348" Jul 24 03:21:59.355698 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:59.355615 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:21:59.418173 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:21:59.418140 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:00.750780 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:00.750749 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:00.811057 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:00.811028 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:01.069649 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:01.069563 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:01.737415 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:01.737384 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:01.802303 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:01.802265 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:02.155256 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:02.155222 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:02.272159 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:02.272126 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:02.351319 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:02.351292 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:05.518759 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:05.518723 2579 patch_prober.go:28] interesting pod/image-registry-77678456c6-kmzrp container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 24 03:22:05.519207 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:05.518792 2579 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-77678456c6-kmzrp" podUID="003896e6-6475-4611-92c7-9beea3b43ae9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 03:22:06.141724 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:06.141673 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:06.233929 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:06.233900 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:06.841319 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:06.841287 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:06.942685 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:06.942658 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:08.983593 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:08.983561 2579 patch_prober.go:28] interesting pod/image-registry-77678456c6-kmzrp container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 24 03:22:08.984029 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:08.983616 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-77678456c6-kmzrp" podUID="003896e6-6475-4611-92c7-9beea3b43ae9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 03:22:12.495746 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:12.495706 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:12.596834 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:12.596799 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:12.654626 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:12.654599 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:12.662679 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:22:12.662656 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-4zq5x" podUID="1a36f9a1-1cc8-4858-be3f-ec11e34eebab" Jul 24 03:22:13.064243 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:13.064208 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4zq5x" Jul 24 03:22:15.518294 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:15.518253 2579 patch_prober.go:28] interesting pod/image-registry-77678456c6-kmzrp container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 24 03:22:15.518886 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:15.518321 2579 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-77678456c6-kmzrp" podUID="003896e6-6475-4611-92c7-9beea3b43ae9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 03:22:15.938280 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:15.938243 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls\") pod \"dns-default-4zq5x\" (UID: \"1a36f9a1-1cc8-4858-be3f-ec11e34eebab\") " pod="openshift-dns/dns-default-4zq5x" Jul 24 03:22:15.941041 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:15.941013 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a36f9a1-1cc8-4858-be3f-ec11e34eebab-metrics-tls\") pod \"dns-default-4zq5x\" (UID: \"1a36f9a1-1cc8-4858-be3f-ec11e34eebab\") " pod="openshift-dns/dns-default-4zq5x" Jul 24 03:22:16.067027 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:16.066996 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fzmf8\"" Jul 24 03:22:16.074889 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:16.074864 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4zq5x" Jul 24 03:22:16.309006 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:16.308857 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4zq5x"] Jul 24 03:22:16.311472 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:22:16.311422 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a36f9a1_1cc8_4858_be3f_ec11e34eebab.slice/crio-f74b40b1ddf5e117dfa582b5db599fc887e4e008a3b54849d3aafe0b4e06b93e WatchSource:0}: Error finding container f74b40b1ddf5e117dfa582b5db599fc887e4e008a3b54849d3aafe0b4e06b93e: Status 404 returned error can't find the container with id f74b40b1ddf5e117dfa582b5db599fc887e4e008a3b54849d3aafe0b4e06b93e Jul 24 03:22:16.846365 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:16.846334 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:17.082214 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:17.082175 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4zq5x" event={"ID":"1a36f9a1-1cc8-4858-be3f-ec11e34eebab","Type":"ContainerStarted","Data":"f74b40b1ddf5e117dfa582b5db599fc887e4e008a3b54849d3aafe0b4e06b93e"} Jul 24 03:22:18.559210 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:18.559129 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:18.985038 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:18.985012 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-77678456c6-kmzrp" Jul 24 03:22:19.091275 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:19.091244 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4zq5x" event={"ID":"1a36f9a1-1cc8-4858-be3f-ec11e34eebab","Type":"ContainerStarted","Data":"926fce1d6bb0d16198aad35f0a47f875099074b3521d096f867547ee92299c78"} Jul 24 03:22:19.091452 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:19.091293 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4zq5x" event={"ID":"1a36f9a1-1cc8-4858-be3f-ec11e34eebab","Type":"ContainerStarted","Data":"ace2c21ca34efa39b3e1c744631441c1f23e19681e016ac712d5db5c3dd5715f"} Jul 24 03:22:19.091452 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:19.091336 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4zq5x" Jul 24 03:22:19.122143 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:19.122088 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4zq5x" podStartSLOduration=252.301523936 podStartE2EDuration="4m14.122069885s" totalImagesPullingTime="1.820545949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:18:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:22:19.121597033 +0000 UTC m=+288.561016496" watchObservedRunningTime="2026-07-24 03:22:19.122069885 +0000 UTC m=+288.561489345" Jul 24 03:22:19.126452 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:19.126428 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:19.221200 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:19.221169 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:19.319045 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:19.318961 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:20.501209 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:20.501174 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:20.555291 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:20.555260 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:20.733810 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:20.733779 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:20.836107 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:20.836030 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:26.322710 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:26.322675 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:26.423635 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:26.423603 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:26.523976 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:26.523946 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:29.097632 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:29.097552 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4zq5x" Jul 24 03:22:29.575378 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:29.575347 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:29.675767 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:29.675739 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:30.480193 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:30.480162 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:30.535774 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:30.535744 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:30.895934 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:30.895903 2579 kubelet.go:1648] "Image garbage collection succeeded" Jul 24 03:22:33.139975 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:33.139939 2579 generic.go:356] "Generic (PLEG): container finished" podID="587fc069-a2aa-4701-82e9-65e01ab74ade" containerID="5cb9f94570650cf345e67830f0ea89eadbb21b54a3b05956840b4a2500d0cc93" exitCode=0 Jul 24 03:22:33.140526 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:33.139991 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-8gpjf" event={"ID":"587fc069-a2aa-4701-82e9-65e01ab74ade","Type":"ContainerDied","Data":"5cb9f94570650cf345e67830f0ea89eadbb21b54a3b05956840b4a2500d0cc93"} Jul 24 03:22:33.140526 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:33.140364 2579 scope.go:117] "RemoveContainer" containerID="5cb9f94570650cf345e67830f0ea89eadbb21b54a3b05956840b4a2500d0cc93" Jul 24 03:22:33.951277 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:33.951247 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4zq5x_1a36f9a1-1cc8-4858-be3f-ec11e34eebab/dns/0.log" Jul 24 03:22:33.957260 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:33.957236 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4zq5x_1a36f9a1-1cc8-4858-be3f-ec11e34eebab/kube-rbac-proxy/0.log" Jul 24 03:22:34.145833 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:34.145802 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-8gpjf" event={"ID":"587fc069-a2aa-4701-82e9-65e01ab74ade","Type":"ContainerStarted","Data":"a4cd9bd23602176d64848e3265f6a8a7668685b92c682a2cc671ec63458510ab"} Jul 24 03:22:34.453792 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:34.453765 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zgsks_a5b9b325-d029-42ca-8dfb-c75c2dd0c726/dns-node-resolver/0.log" Jul 24 03:22:36.958971 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:36.958918 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:37.011173 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:37.011141 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:38.161502 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:38.161443 2579 generic.go:356] "Generic (PLEG): container finished" podID="0d716e5b-e403-4cf6-94f4-8dbca03f5e25" containerID="7f311605b7a6a5b4c94fc2b1aff76a6802529c6b9ee7446331b280943cda6c1f" exitCode=0 Jul 24 03:22:38.161863 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:38.161505 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q5l2j" event={"ID":"0d716e5b-e403-4cf6-94f4-8dbca03f5e25","Type":"ContainerDied","Data":"7f311605b7a6a5b4c94fc2b1aff76a6802529c6b9ee7446331b280943cda6c1f"} Jul 24 03:22:38.161906 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:38.161874 2579 scope.go:117] "RemoveContainer" containerID="7f311605b7a6a5b4c94fc2b1aff76a6802529c6b9ee7446331b280943cda6c1f" Jul 24 03:22:39.166624 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:39.166590 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-q5l2j" event={"ID":"0d716e5b-e403-4cf6-94f4-8dbca03f5e25","Type":"ContainerStarted","Data":"a308d8d5bf03ef6e8c9f885b5d27580a53a5d354f271d635b68a981f4cb5d6e5"} Jul 24 03:22:45.482086 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:45.482047 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:45.537481 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:45.537388 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:45.969112 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:45.969077 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:22:46.068536 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:22:46.068491 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:23:15.145668 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:15.145636 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:23:15.243098 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:15.243066 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:23:20.223510 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.223475 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-58469cd6d-xrpbr"] Jul 24 03:23:20.226922 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.226907 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" Jul 24 03:23:20.229093 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.229073 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rnw2n\"" Jul 24 03:23:20.229227 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.229208 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 24 03:23:20.229498 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.229478 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 24 03:23:20.229585 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.229505 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 24 03:23:20.229585 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.229549 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 24 03:23:20.229585 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.229509 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 24 03:23:20.233942 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.233604 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 24 03:23:20.237936 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.237917 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-58469cd6d-xrpbr"] Jul 24 03:23:20.308829 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.308790 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1c880c3-5228-4b9d-a829-04cb5c44f37b-serving-certs-ca-bundle\") pod \"telemeter-client-58469cd6d-xrpbr\" (UID: \"d1c880c3-5228-4b9d-a829-04cb5c44f37b\") " pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" Jul 24 03:23:20.309009 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.308840 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d1c880c3-5228-4b9d-a829-04cb5c44f37b-telemeter-client-tls\") pod \"telemeter-client-58469cd6d-xrpbr\" (UID: \"d1c880c3-5228-4b9d-a829-04cb5c44f37b\") " pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" Jul 24 03:23:20.309009 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.308894 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htz2n\" (UniqueName: \"kubernetes.io/projected/d1c880c3-5228-4b9d-a829-04cb5c44f37b-kube-api-access-htz2n\") pod \"telemeter-client-58469cd6d-xrpbr\" (UID: \"d1c880c3-5228-4b9d-a829-04cb5c44f37b\") " pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" Jul 24 03:23:20.309009 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.308916 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1c880c3-5228-4b9d-a829-04cb5c44f37b-metrics-client-ca\") pod \"telemeter-client-58469cd6d-xrpbr\" (UID: \"d1c880c3-5228-4b9d-a829-04cb5c44f37b\") " pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" Jul 24 03:23:20.309009 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.308940 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d1c880c3-5228-4b9d-a829-04cb5c44f37b-federate-client-tls\") pod \"telemeter-client-58469cd6d-xrpbr\" (UID: \"d1c880c3-5228-4b9d-a829-04cb5c44f37b\") " pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" Jul 24 03:23:20.309009 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.308991 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1c880c3-5228-4b9d-a829-04cb5c44f37b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-58469cd6d-xrpbr\" (UID: \"d1c880c3-5228-4b9d-a829-04cb5c44f37b\") " pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" Jul 24 03:23:20.309167 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.309021 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1c880c3-5228-4b9d-a829-04cb5c44f37b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-58469cd6d-xrpbr\" (UID: \"d1c880c3-5228-4b9d-a829-04cb5c44f37b\") " pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" Jul 24 03:23:20.309167 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.309057 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d1c880c3-5228-4b9d-a829-04cb5c44f37b-secret-telemeter-client\") pod \"telemeter-client-58469cd6d-xrpbr\" (UID: \"d1c880c3-5228-4b9d-a829-04cb5c44f37b\") " pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" Jul 24 03:23:20.409756 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.409721 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d1c880c3-5228-4b9d-a829-04cb5c44f37b-secret-telemeter-client\") pod \"telemeter-client-58469cd6d-xrpbr\" (UID: \"d1c880c3-5228-4b9d-a829-04cb5c44f37b\") " pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" Jul 24 03:23:20.409930 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.409763 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1c880c3-5228-4b9d-a829-04cb5c44f37b-serving-certs-ca-bundle\") pod \"telemeter-client-58469cd6d-xrpbr\" (UID: \"d1c880c3-5228-4b9d-a829-04cb5c44f37b\") " pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" Jul 24 03:23:20.409930 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.409889 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d1c880c3-5228-4b9d-a829-04cb5c44f37b-telemeter-client-tls\") pod \"telemeter-client-58469cd6d-xrpbr\" (UID: \"d1c880c3-5228-4b9d-a829-04cb5c44f37b\") " pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" Jul 24 03:23:20.409930 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.409929 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htz2n\" (UniqueName: \"kubernetes.io/projected/d1c880c3-5228-4b9d-a829-04cb5c44f37b-kube-api-access-htz2n\") pod \"telemeter-client-58469cd6d-xrpbr\" (UID: \"d1c880c3-5228-4b9d-a829-04cb5c44f37b\") " pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" Jul 24 03:23:20.410060 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.409948 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1c880c3-5228-4b9d-a829-04cb5c44f37b-metrics-client-ca\") pod \"telemeter-client-58469cd6d-xrpbr\" (UID: \"d1c880c3-5228-4b9d-a829-04cb5c44f37b\") " pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" Jul 24 03:23:20.410060 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.409974 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d1c880c3-5228-4b9d-a829-04cb5c44f37b-federate-client-tls\") pod \"telemeter-client-58469cd6d-xrpbr\" (UID: \"d1c880c3-5228-4b9d-a829-04cb5c44f37b\") " pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" Jul 24 03:23:20.410060 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.410021 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1c880c3-5228-4b9d-a829-04cb5c44f37b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-58469cd6d-xrpbr\" (UID: \"d1c880c3-5228-4b9d-a829-04cb5c44f37b\") " pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" Jul 24 03:23:20.410060 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.410047 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1c880c3-5228-4b9d-a829-04cb5c44f37b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-58469cd6d-xrpbr\" (UID: \"d1c880c3-5228-4b9d-a829-04cb5c44f37b\") " pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" Jul 24 03:23:20.410595 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.410566 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1c880c3-5228-4b9d-a829-04cb5c44f37b-serving-certs-ca-bundle\") pod \"telemeter-client-58469cd6d-xrpbr\" (UID: \"d1c880c3-5228-4b9d-a829-04cb5c44f37b\") " pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" Jul 24 03:23:20.410704 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.410639 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1c880c3-5228-4b9d-a829-04cb5c44f37b-metrics-client-ca\") pod \"telemeter-client-58469cd6d-xrpbr\" (UID: \"d1c880c3-5228-4b9d-a829-04cb5c44f37b\") " pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" Jul 24 03:23:20.411127 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.411104 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1c880c3-5228-4b9d-a829-04cb5c44f37b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-58469cd6d-xrpbr\" (UID: \"d1c880c3-5228-4b9d-a829-04cb5c44f37b\") " pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" Jul 24 03:23:20.412610 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.412585 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1c880c3-5228-4b9d-a829-04cb5c44f37b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-58469cd6d-xrpbr\" (UID: \"d1c880c3-5228-4b9d-a829-04cb5c44f37b\") " pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" Jul 24 03:23:20.412873 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.412855 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d1c880c3-5228-4b9d-a829-04cb5c44f37b-telemeter-client-tls\") pod \"telemeter-client-58469cd6d-xrpbr\" (UID: \"d1c880c3-5228-4b9d-a829-04cb5c44f37b\") " pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" Jul 24 03:23:20.412938 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.412917 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d1c880c3-5228-4b9d-a829-04cb5c44f37b-federate-client-tls\") pod \"telemeter-client-58469cd6d-xrpbr\" (UID: \"d1c880c3-5228-4b9d-a829-04cb5c44f37b\") " pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" Jul 24 03:23:20.412938 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.412921 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d1c880c3-5228-4b9d-a829-04cb5c44f37b-secret-telemeter-client\") pod \"telemeter-client-58469cd6d-xrpbr\" (UID: \"d1c880c3-5228-4b9d-a829-04cb5c44f37b\") " pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" Jul 24 03:23:20.418056 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.418034 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htz2n\" (UniqueName: \"kubernetes.io/projected/d1c880c3-5228-4b9d-a829-04cb5c44f37b-kube-api-access-htz2n\") pod \"telemeter-client-58469cd6d-xrpbr\" (UID: \"d1c880c3-5228-4b9d-a829-04cb5c44f37b\") " pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" Jul 24 03:23:20.536772 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.536682 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" Jul 24 03:23:20.669552 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.669527 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-58469cd6d-xrpbr"] Jul 24 03:23:20.671895 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:23:20.671867 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c880c3_5228_4b9d_a829_04cb5c44f37b.slice/crio-ae58f3a9695b8ec3f81d3601f29826835750c152594511e52aaa9e268e4b6139 WatchSource:0}: Error finding container ae58f3a9695b8ec3f81d3601f29826835750c152594511e52aaa9e268e4b6139: Status 404 returned error can't find the container with id ae58f3a9695b8ec3f81d3601f29826835750c152594511e52aaa9e268e4b6139 Jul 24 03:23:20.673643 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:20.673628 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 03:23:21.321646 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:21.321603 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" event={"ID":"d1c880c3-5228-4b9d-a829-04cb5c44f37b","Type":"ContainerStarted","Data":"ae58f3a9695b8ec3f81d3601f29826835750c152594511e52aaa9e268e4b6139"} Jul 24 03:23:23.332148 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:23.332108 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" event={"ID":"d1c880c3-5228-4b9d-a829-04cb5c44f37b","Type":"ContainerStarted","Data":"bf56c07cdcf0d29ec4f8d94aaabe0860de336f53a20ee3ff267c334d7c832984"} Jul 24 03:23:24.337728 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:24.337693 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" event={"ID":"d1c880c3-5228-4b9d-a829-04cb5c44f37b","Type":"ContainerStarted","Data":"5c408870394b461bcd5e2f0c0b8629abbde036f52aebf1538eb31ddf53175476"} Jul 24 03:23:24.338165 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:24.337748 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" event={"ID":"d1c880c3-5228-4b9d-a829-04cb5c44f37b","Type":"ContainerStarted","Data":"0019d5b746c3bf7b331051fe1a5ede995db98edc476f797120f4e20b97cd853f"} Jul 24 03:23:24.364760 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:24.364711 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-58469cd6d-xrpbr" podStartSLOduration=1.629406308 podStartE2EDuration="4.364696064s" totalImagesPullingTime="2.735289756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:23:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:23:24.36337402 +0000 UTC m=+353.802793480" watchObservedRunningTime="2026-07-24 03:23:24.364696064 +0000 UTC m=+353.804115523" Jul 24 03:23:25.331110 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:25.331076 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:23:25.427378 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:25.427335 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:23:29.048955 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:29.048924 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:23:29.112519 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:29.112485 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:23:36.111127 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:23:36.111091 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:24:04.131440 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:24:04.131403 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:24:04.231680 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:24:04.231648 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:24:30.447812 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:24:30.447776 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:24:30.621389 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:24:30.621353 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:24:42.429153 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:24:42.429096 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:24:42.529304 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:24:42.529271 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:24:45.444421 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:24:45.444372 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:24:45.537896 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:24:45.537859 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:24:45.653596 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:24:45.653556 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:24:56.403416 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:24:56.403333 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:24:56.458975 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:24:56.458939 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:24:56.519335 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:24:56.519274 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:25:06.610507 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:06.610476 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:25:06.705737 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:06.705708 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:25:28.373683 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:28.373650 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:25:29.151101 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:29.151066 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:25:31.989952 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:31.989907 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:25:32.092179 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:32.092139 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:25:33.825907 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:33.825855 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:25:33.930825 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:33.930786 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:25:34.363088 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:34.363055 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:25:34.465867 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:34.465836 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:25:36.678495 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:36.678444 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:25:36.888986 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:36.888950 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:25:38.981922 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:38.981890 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:25:39.034357 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:39.034324 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:25:41.363776 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:41.363748 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:25:41.418230 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:41.418196 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:25:44.291269 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:44.291238 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:25:46.444161 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:46.444121 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:25:46.513150 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:46.513122 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:25:47.882342 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:47.882308 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-27zv2"] Jul 24 03:25:47.885524 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:47.885503 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-27zv2" Jul 24 03:25:47.887594 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:47.887576 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 24 03:25:47.888350 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:47.888335 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-srq55\"" Jul 24 03:25:47.888429 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:47.888383 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 24 03:25:47.895669 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:47.895642 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-27zv2"] Jul 24 03:25:48.032136 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:48.032097 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xv4f\" (UniqueName: \"kubernetes.io/projected/8187f663-7660-481e-a5d7-4f5b1d6a5179-kube-api-access-9xv4f\") pod \"openshift-lws-operator-fd8ccff4c-27zv2\" (UID: \"8187f663-7660-481e-a5d7-4f5b1d6a5179\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-27zv2" Jul 24 03:25:48.032297 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:48.032144 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8187f663-7660-481e-a5d7-4f5b1d6a5179-tmp\") pod \"openshift-lws-operator-fd8ccff4c-27zv2\" (UID: \"8187f663-7660-481e-a5d7-4f5b1d6a5179\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-27zv2" Jul 24 03:25:48.133100 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:48.133012 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xv4f\" (UniqueName: \"kubernetes.io/projected/8187f663-7660-481e-a5d7-4f5b1d6a5179-kube-api-access-9xv4f\") pod \"openshift-lws-operator-fd8ccff4c-27zv2\" (UID: \"8187f663-7660-481e-a5d7-4f5b1d6a5179\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-27zv2" Jul 24 03:25:48.133100 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:48.133056 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8187f663-7660-481e-a5d7-4f5b1d6a5179-tmp\") pod \"openshift-lws-operator-fd8ccff4c-27zv2\" (UID: \"8187f663-7660-481e-a5d7-4f5b1d6a5179\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-27zv2" Jul 24 03:25:48.133383 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:48.133369 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8187f663-7660-481e-a5d7-4f5b1d6a5179-tmp\") pod \"openshift-lws-operator-fd8ccff4c-27zv2\" (UID: \"8187f663-7660-481e-a5d7-4f5b1d6a5179\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-27zv2" Jul 24 03:25:48.141645 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:48.141615 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xv4f\" (UniqueName: \"kubernetes.io/projected/8187f663-7660-481e-a5d7-4f5b1d6a5179-kube-api-access-9xv4f\") pod \"openshift-lws-operator-fd8ccff4c-27zv2\" (UID: \"8187f663-7660-481e-a5d7-4f5b1d6a5179\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-27zv2" Jul 24 03:25:48.194210 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:48.194170 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-27zv2" Jul 24 03:25:48.327040 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:48.327011 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-27zv2"] Jul 24 03:25:48.329967 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:25:48.329940 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8187f663_7660_481e_a5d7_4f5b1d6a5179.slice/crio-6f2f6f2405b05975a3abf3aa9a3c09ecc4ab50ac8e68644e8c8dfb110476c63b WatchSource:0}: Error finding container 6f2f6f2405b05975a3abf3aa9a3c09ecc4ab50ac8e68644e8c8dfb110476c63b: Status 404 returned error can't find the container with id 6f2f6f2405b05975a3abf3aa9a3c09ecc4ab50ac8e68644e8c8dfb110476c63b Jul 24 03:25:48.841096 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:48.841060 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-27zv2" event={"ID":"8187f663-7660-481e-a5d7-4f5b1d6a5179","Type":"ContainerStarted","Data":"6f2f6f2405b05975a3abf3aa9a3c09ecc4ab50ac8e68644e8c8dfb110476c63b"} Jul 24 03:25:54.865777 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:54.865742 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-27zv2" event={"ID":"8187f663-7660-481e-a5d7-4f5b1d6a5179","Type":"ContainerStarted","Data":"356132910a9b7cb5f9126f725ff5068b1c24cee07519f7038f40178e689f7252"} Jul 24 03:25:54.882886 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:54.882835 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-27zv2" podStartSLOduration=2.020842391 podStartE2EDuration="7.882822224s" totalImagesPullingTime="5.861979833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:25:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:25:54.881913862 +0000 UTC m=+504.321333323" watchObservedRunningTime="2026-07-24 03:25:54.882822224 +0000 UTC m=+504.322241685" Jul 24 03:25:55.686767 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:55.686724 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:25:55.796903 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:55.796873 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:25:57.615775 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:57.615741 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:25:57.717356 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:57.717314 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:25:57.803060 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:25:57.803024 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:09.518540 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:09.518504 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:09.684212 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:09.684184 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:10.124202 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:10.124170 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:10.594559 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:10.594447 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:10.797723 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:10.797680 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:10.950725 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:10.950688 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:10.970621 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:10.970582 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67f4587fcc-44l4m"] Jul 24 03:26:10.977056 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:10.977039 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67f4587fcc-44l4m" Jul 24 03:26:10.985200 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:10.985174 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 24 03:26:10.985329 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:10.985216 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 24 03:26:10.985329 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:10.985234 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 24 03:26:10.985329 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:10.985234 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 24 03:26:10.985329 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:10.985262 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-q7bbg\"" Jul 24 03:26:10.998350 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:10.998325 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vczb8\" (UniqueName: \"kubernetes.io/projected/56b90bb3-b363-4be6-ba7c-6c5c10ff8b06-kube-api-access-vczb8\") pod \"opendatahub-operator-controller-manager-67f4587fcc-44l4m\" (UID: \"56b90bb3-b363-4be6-ba7c-6c5c10ff8b06\") " pod="opendatahub/opendatahub-operator-controller-manager-67f4587fcc-44l4m" Jul 24 03:26:10.998491 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:10.998371 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/56b90bb3-b363-4be6-ba7c-6c5c10ff8b06-manifests\") pod \"opendatahub-operator-controller-manager-67f4587fcc-44l4m\" (UID: \"56b90bb3-b363-4be6-ba7c-6c5c10ff8b06\") " pod="opendatahub/opendatahub-operator-controller-manager-67f4587fcc-44l4m" Jul 24 03:26:10.998491 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:10.998406 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56b90bb3-b363-4be6-ba7c-6c5c10ff8b06-tmp\") pod \"opendatahub-operator-controller-manager-67f4587fcc-44l4m\" (UID: \"56b90bb3-b363-4be6-ba7c-6c5c10ff8b06\") " pod="opendatahub/opendatahub-operator-controller-manager-67f4587fcc-44l4m" Jul 24 03:26:10.998491 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:10.998480 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/56b90bb3-b363-4be6-ba7c-6c5c10ff8b06-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67f4587fcc-44l4m\" (UID: \"56b90bb3-b363-4be6-ba7c-6c5c10ff8b06\") " pod="opendatahub/opendatahub-operator-controller-manager-67f4587fcc-44l4m" Jul 24 03:26:10.998610 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:10.998510 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56b90bb3-b363-4be6-ba7c-6c5c10ff8b06-webhook-cert\") pod \"opendatahub-operator-controller-manager-67f4587fcc-44l4m\" (UID: \"56b90bb3-b363-4be6-ba7c-6c5c10ff8b06\") " pod="opendatahub/opendatahub-operator-controller-manager-67f4587fcc-44l4m" Jul 24 03:26:11.024964 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:11.024936 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67f4587fcc-44l4m"] Jul 24 03:26:11.099731 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:11.099704 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56b90bb3-b363-4be6-ba7c-6c5c10ff8b06-tmp\") pod \"opendatahub-operator-controller-manager-67f4587fcc-44l4m\" (UID: \"56b90bb3-b363-4be6-ba7c-6c5c10ff8b06\") " pod="opendatahub/opendatahub-operator-controller-manager-67f4587fcc-44l4m" Jul 24 03:26:11.099908 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:11.099742 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/56b90bb3-b363-4be6-ba7c-6c5c10ff8b06-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67f4587fcc-44l4m\" (UID: \"56b90bb3-b363-4be6-ba7c-6c5c10ff8b06\") " pod="opendatahub/opendatahub-operator-controller-manager-67f4587fcc-44l4m" Jul 24 03:26:11.099908 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:11.099762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56b90bb3-b363-4be6-ba7c-6c5c10ff8b06-webhook-cert\") pod \"opendatahub-operator-controller-manager-67f4587fcc-44l4m\" (UID: \"56b90bb3-b363-4be6-ba7c-6c5c10ff8b06\") " pod="opendatahub/opendatahub-operator-controller-manager-67f4587fcc-44l4m" Jul 24 03:26:11.099908 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:11.099786 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vczb8\" (UniqueName: \"kubernetes.io/projected/56b90bb3-b363-4be6-ba7c-6c5c10ff8b06-kube-api-access-vczb8\") pod \"opendatahub-operator-controller-manager-67f4587fcc-44l4m\" (UID: \"56b90bb3-b363-4be6-ba7c-6c5c10ff8b06\") " pod="opendatahub/opendatahub-operator-controller-manager-67f4587fcc-44l4m" Jul 24 03:26:11.099908 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:11.099832 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/56b90bb3-b363-4be6-ba7c-6c5c10ff8b06-manifests\") pod \"opendatahub-operator-controller-manager-67f4587fcc-44l4m\" (UID: \"56b90bb3-b363-4be6-ba7c-6c5c10ff8b06\") " pod="opendatahub/opendatahub-operator-controller-manager-67f4587fcc-44l4m" Jul 24 03:26:11.100224 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:11.100198 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/56b90bb3-b363-4be6-ba7c-6c5c10ff8b06-manifests\") pod \"opendatahub-operator-controller-manager-67f4587fcc-44l4m\" (UID: \"56b90bb3-b363-4be6-ba7c-6c5c10ff8b06\") " pod="opendatahub/opendatahub-operator-controller-manager-67f4587fcc-44l4m" Jul 24 03:26:11.102147 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:11.102120 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56b90bb3-b363-4be6-ba7c-6c5c10ff8b06-tmp\") pod \"opendatahub-operator-controller-manager-67f4587fcc-44l4m\" (UID: \"56b90bb3-b363-4be6-ba7c-6c5c10ff8b06\") " pod="opendatahub/opendatahub-operator-controller-manager-67f4587fcc-44l4m" Jul 24 03:26:11.102256 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:11.102239 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56b90bb3-b363-4be6-ba7c-6c5c10ff8b06-webhook-cert\") pod \"opendatahub-operator-controller-manager-67f4587fcc-44l4m\" (UID: \"56b90bb3-b363-4be6-ba7c-6c5c10ff8b06\") " pod="opendatahub/opendatahub-operator-controller-manager-67f4587fcc-44l4m" Jul 24 03:26:11.102443 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:11.102421 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/56b90bb3-b363-4be6-ba7c-6c5c10ff8b06-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67f4587fcc-44l4m\" (UID: \"56b90bb3-b363-4be6-ba7c-6c5c10ff8b06\") " pod="opendatahub/opendatahub-operator-controller-manager-67f4587fcc-44l4m" Jul 24 03:26:11.117306 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:11.117267 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vczb8\" (UniqueName: \"kubernetes.io/projected/56b90bb3-b363-4be6-ba7c-6c5c10ff8b06-kube-api-access-vczb8\") pod \"opendatahub-operator-controller-manager-67f4587fcc-44l4m\" (UID: \"56b90bb3-b363-4be6-ba7c-6c5c10ff8b06\") " pod="opendatahub/opendatahub-operator-controller-manager-67f4587fcc-44l4m" Jul 24 03:26:11.287441 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:11.287360 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67f4587fcc-44l4m" Jul 24 03:26:11.353267 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:11.353178 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:11.445295 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:11.445254 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67f4587fcc-44l4m"] Jul 24 03:26:11.448451 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:26:11.448417 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b90bb3_b363_4be6_ba7c_6c5c10ff8b06.slice/crio-6dc2f09f5b52a647529bd0f4eed2e9ed4e8cd8de4759bc487a791bcb36f0b732 WatchSource:0}: Error finding container 6dc2f09f5b52a647529bd0f4eed2e9ed4e8cd8de4759bc487a791bcb36f0b732: Status 404 returned error can't find the container with id 6dc2f09f5b52a647529bd0f4eed2e9ed4e8cd8de4759bc487a791bcb36f0b732 Jul 24 03:26:11.478102 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:11.478075 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:11.614007 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:11.613980 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:11.793628 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:11.793589 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:11.928927 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:11.928835 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67f4587fcc-44l4m" event={"ID":"56b90bb3-b363-4be6-ba7c-6c5c10ff8b06","Type":"ContainerStarted","Data":"6dc2f09f5b52a647529bd0f4eed2e9ed4e8cd8de4759bc487a791bcb36f0b732"} Jul 24 03:26:11.936515 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:11.936485 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:12.043232 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:12.043196 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:14.942141 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:14.942102 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67f4587fcc-44l4m" event={"ID":"56b90bb3-b363-4be6-ba7c-6c5c10ff8b06","Type":"ContainerStarted","Data":"1ff19718462447e906f8fbb4c9a22dd502d18ee6ef65abc048876f7eb745f718"} Jul 24 03:26:15.947762 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:15.947726 2579 generic.go:356] "Generic (PLEG): container finished" podID="56b90bb3-b363-4be6-ba7c-6c5c10ff8b06" containerID="1ff19718462447e906f8fbb4c9a22dd502d18ee6ef65abc048876f7eb745f718" exitCode=0 Jul 24 03:26:15.948136 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:15.947797 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67f4587fcc-44l4m" event={"ID":"56b90bb3-b363-4be6-ba7c-6c5c10ff8b06","Type":"ContainerDied","Data":"1ff19718462447e906f8fbb4c9a22dd502d18ee6ef65abc048876f7eb745f718"} Jul 24 03:26:16.952965 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:16.952931 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67f4587fcc-44l4m" event={"ID":"56b90bb3-b363-4be6-ba7c-6c5c10ff8b06","Type":"ContainerStarted","Data":"1105eaf64ad69a9a7184e0f0db487f7c95dc6d8d01b4282ccd19a4d0298a54ba"} Jul 24 03:26:16.953442 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:16.953045 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67f4587fcc-44l4m" Jul 24 03:26:16.998084 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:16.998033 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67f4587fcc-44l4m" podStartSLOduration=2.7151815150000003 podStartE2EDuration="6.99801561s" totalImagesPullingTime="3.286783817s" totalInitContainerRuntime="996.050278ms" isStatefulPod=false podCreationTimestamp="2026-07-24 03:26:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:26:16.99709603 +0000 UTC m=+526.436515490" watchObservedRunningTime="2026-07-24 03:26:16.99801561 +0000 UTC m=+526.437435070" Jul 24 03:26:22.204654 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:22.204557 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:22.319245 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:22.319212 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:24.244680 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:24.244634 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:27.958614 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:27.958582 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67f4587fcc-44l4m" Jul 24 03:26:29.195715 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:29.195688 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:29.336361 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:29.336334 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:31.634400 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:31.634366 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:31.933486 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:31.933366 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:32.575362 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:32.575329 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:32.600504 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:32.600478 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5f459d4668-c22g9"] Jul 24 03:26:32.605332 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:32.605315 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f459d4668-c22g9" Jul 24 03:26:32.612745 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:32.612725 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 24 03:26:32.612946 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:32.612929 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 24 03:26:32.613426 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:26:32.613404 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-auth-proxy-tls\" is forbidden: User \"system:node:ip-10-0-134-147.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-134-147.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" type="*v1.Secret" Jul 24 03:26:32.613695 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:26:32.613665 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kube-auth-proxy-5f459d4668-c22g9\" is forbidden: User \"system:node:ip-10-0-134-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-134-147.ec2.internal' and this object" podUID="b5b6f8d8-a71c-4088-bc54-52634f3d8eee" pod="openshift-ingress/kube-auth-proxy-5f459d4668-c22g9" Jul 24 03:26:32.613786 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:26:32.613699 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-auth-proxy-creds\" is forbidden: User \"system:node:ip-10-0-134-147.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-134-147.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" type="*v1.Secret" Jul 24 03:26:32.649185 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:32.649153 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f459d4668-c22g9"] Jul 24 03:26:32.678622 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:32.678590 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp4z2\" (UniqueName: \"kubernetes.io/projected/b5b6f8d8-a71c-4088-bc54-52634f3d8eee-kube-api-access-fp4z2\") pod \"kube-auth-proxy-5f459d4668-c22g9\" (UID: \"b5b6f8d8-a71c-4088-bc54-52634f3d8eee\") " pod="openshift-ingress/kube-auth-proxy-5f459d4668-c22g9" Jul 24 03:26:32.678808 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:32.678648 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5b6f8d8-a71c-4088-bc54-52634f3d8eee-tls-certs\") pod \"kube-auth-proxy-5f459d4668-c22g9\" (UID: \"b5b6f8d8-a71c-4088-bc54-52634f3d8eee\") " pod="openshift-ingress/kube-auth-proxy-5f459d4668-c22g9" Jul 24 03:26:32.678808 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:32.678735 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5b6f8d8-a71c-4088-bc54-52634f3d8eee-tmp\") pod \"kube-auth-proxy-5f459d4668-c22g9\" (UID: \"b5b6f8d8-a71c-4088-bc54-52634f3d8eee\") " pod="openshift-ingress/kube-auth-proxy-5f459d4668-c22g9" Jul 24 03:26:32.779158 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:32.779117 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5b6f8d8-a71c-4088-bc54-52634f3d8eee-tmp\") pod \"kube-auth-proxy-5f459d4668-c22g9\" (UID: \"b5b6f8d8-a71c-4088-bc54-52634f3d8eee\") " pod="openshift-ingress/kube-auth-proxy-5f459d4668-c22g9" Jul 24 03:26:32.779305 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:32.779185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp4z2\" (UniqueName: \"kubernetes.io/projected/b5b6f8d8-a71c-4088-bc54-52634f3d8eee-kube-api-access-fp4z2\") pod \"kube-auth-proxy-5f459d4668-c22g9\" (UID: \"b5b6f8d8-a71c-4088-bc54-52634f3d8eee\") " pod="openshift-ingress/kube-auth-proxy-5f459d4668-c22g9" Jul 24 03:26:32.779305 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:32.779238 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5b6f8d8-a71c-4088-bc54-52634f3d8eee-tls-certs\") pod \"kube-auth-proxy-5f459d4668-c22g9\" (UID: \"b5b6f8d8-a71c-4088-bc54-52634f3d8eee\") " pod="openshift-ingress/kube-auth-proxy-5f459d4668-c22g9" Jul 24 03:26:32.781686 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:32.781655 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5b6f8d8-a71c-4088-bc54-52634f3d8eee-tmp\") pod \"kube-auth-proxy-5f459d4668-c22g9\" (UID: \"b5b6f8d8-a71c-4088-bc54-52634f3d8eee\") " pod="openshift-ingress/kube-auth-proxy-5f459d4668-c22g9" Jul 24 03:26:32.821298 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:32.821258 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp4z2\" (UniqueName: \"kubernetes.io/projected/b5b6f8d8-a71c-4088-bc54-52634f3d8eee-kube-api-access-fp4z2\") pod \"kube-auth-proxy-5f459d4668-c22g9\" (UID: \"b5b6f8d8-a71c-4088-bc54-52634f3d8eee\") " pod="openshift-ingress/kube-auth-proxy-5f459d4668-c22g9" Jul 24 03:26:32.961248 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:32.961213 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:33.120302 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:33.120269 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9cn76"] Jul 24 03:26:33.123647 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:33.123629 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9cn76" Jul 24 03:26:33.126873 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:33.126856 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-4hgg4\"" Jul 24 03:26:33.130704 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:33.130688 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 24 03:26:33.179036 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:33.179003 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9cn76"] Jul 24 03:26:33.183099 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:33.183069 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc7f9\" (UniqueName: \"kubernetes.io/projected/aa9efd80-0fe0-4b2a-aff1-1a516a4dcc2b-kube-api-access-rc7f9\") pod \"odh-model-controller-7998c67dc4-9cn76\" (UID: \"aa9efd80-0fe0-4b2a-aff1-1a516a4dcc2b\") " pod="opendatahub/odh-model-controller-7998c67dc4-9cn76" Jul 24 03:26:33.183247 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:33.183140 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa9efd80-0fe0-4b2a-aff1-1a516a4dcc2b-cert\") pod \"odh-model-controller-7998c67dc4-9cn76\" (UID: \"aa9efd80-0fe0-4b2a-aff1-1a516a4dcc2b\") " pod="opendatahub/odh-model-controller-7998c67dc4-9cn76" Jul 24 03:26:33.259895 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:33.259816 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:33.284238 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:33.284206 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc7f9\" (UniqueName: \"kubernetes.io/projected/aa9efd80-0fe0-4b2a-aff1-1a516a4dcc2b-kube-api-access-rc7f9\") pod \"odh-model-controller-7998c67dc4-9cn76\" (UID: \"aa9efd80-0fe0-4b2a-aff1-1a516a4dcc2b\") " pod="opendatahub/odh-model-controller-7998c67dc4-9cn76" Jul 24 03:26:33.284411 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:33.284275 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa9efd80-0fe0-4b2a-aff1-1a516a4dcc2b-cert\") pod \"odh-model-controller-7998c67dc4-9cn76\" (UID: \"aa9efd80-0fe0-4b2a-aff1-1a516a4dcc2b\") " pod="opendatahub/odh-model-controller-7998c67dc4-9cn76" Jul 24 03:26:33.284501 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:26:33.284412 2579 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 24 03:26:33.284501 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:26:33.284497 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa9efd80-0fe0-4b2a-aff1-1a516a4dcc2b-cert podName:aa9efd80-0fe0-4b2a-aff1-1a516a4dcc2b nodeName:}" failed. No retries permitted until 2026-07-24 03:26:33.784452808 +0000 UTC m=+543.223872251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/aa9efd80-0fe0-4b2a-aff1-1a516a4dcc2b-cert") pod "odh-model-controller-7998c67dc4-9cn76" (UID: "aa9efd80-0fe0-4b2a-aff1-1a516a4dcc2b") : secret "odh-model-controller-webhook-cert" not found Jul 24 03:26:33.316630 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:33.316598 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc7f9\" (UniqueName: \"kubernetes.io/projected/aa9efd80-0fe0-4b2a-aff1-1a516a4dcc2b-kube-api-access-rc7f9\") pod \"odh-model-controller-7998c67dc4-9cn76\" (UID: \"aa9efd80-0fe0-4b2a-aff1-1a516a4dcc2b\") " pod="opendatahub/odh-model-controller-7998c67dc4-9cn76" Jul 24 03:26:33.617242 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:33.617210 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:33.719025 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:33.718986 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 24 03:26:33.722405 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:33.722382 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5b6f8d8-a71c-4088-bc54-52634f3d8eee-tls-certs\") pod \"kube-auth-proxy-5f459d4668-c22g9\" (UID: \"b5b6f8d8-a71c-4088-bc54-52634f3d8eee\") " pod="openshift-ingress/kube-auth-proxy-5f459d4668-c22g9" Jul 24 03:26:33.789073 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:33.789033 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa9efd80-0fe0-4b2a-aff1-1a516a4dcc2b-cert\") pod \"odh-model-controller-7998c67dc4-9cn76\" (UID: \"aa9efd80-0fe0-4b2a-aff1-1a516a4dcc2b\") " pod="opendatahub/odh-model-controller-7998c67dc4-9cn76" Jul 24 03:26:33.791523 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:33.791502 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa9efd80-0fe0-4b2a-aff1-1a516a4dcc2b-cert\") pod \"odh-model-controller-7998c67dc4-9cn76\" (UID: \"aa9efd80-0fe0-4b2a-aff1-1a516a4dcc2b\") " pod="opendatahub/odh-model-controller-7998c67dc4-9cn76" Jul 24 03:26:33.815442 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:33.815408 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f459d4668-c22g9" Jul 24 03:26:34.017305 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:34.017227 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f459d4668-c22g9"] Jul 24 03:26:34.021659 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:26:34.021633 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b6f8d8_a71c_4088_bc54_52634f3d8eee.slice/crio-18f118b7dabb4e438ccea30120eb4a4e98c235f3113a1213bccf4294faeae3fe WatchSource:0}: Error finding container 18f118b7dabb4e438ccea30120eb4a4e98c235f3113a1213bccf4294faeae3fe: Status 404 returned error can't find the container with id 18f118b7dabb4e438ccea30120eb4a4e98c235f3113a1213bccf4294faeae3fe Jul 24 03:26:34.033144 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:34.033123 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9cn76" Jul 24 03:26:34.121478 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:34.121431 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 24 03:26:34.187453 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:34.187375 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9cn76"] Jul 24 03:26:34.191192 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:26:34.191150 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa9efd80_0fe0_4b2a_aff1_1a516a4dcc2b.slice/crio-063dabf7c0812d538fa3fae5b068ba620f9788f48a3f630e5d50b1baf4cf5233 WatchSource:0}: Error finding container 063dabf7c0812d538fa3fae5b068ba620f9788f48a3f630e5d50b1baf4cf5233: Status 404 returned error can't find the container with id 063dabf7c0812d538fa3fae5b068ba620f9788f48a3f630e5d50b1baf4cf5233 Jul 24 03:26:34.632005 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:34.631968 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:34.633714 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:34.633690 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-cngtc"] Jul 24 03:26:34.636921 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:34.636899 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-cngtc" Jul 24 03:26:34.639657 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:34.639632 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 24 03:26:34.658291 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:34.658265 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-cngtc"] Jul 24 03:26:34.698232 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:34.698180 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56d05014-f053-447d-b1d2-37b530a3ef52-tls-certs\") pod \"model-serving-api-54d89bd5fc-cngtc\" (UID: \"56d05014-f053-447d-b1d2-37b530a3ef52\") " pod="opendatahub/model-serving-api-54d89bd5fc-cngtc" Jul 24 03:26:34.698385 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:34.698245 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h7z8\" (UniqueName: \"kubernetes.io/projected/56d05014-f053-447d-b1d2-37b530a3ef52-kube-api-access-8h7z8\") pod \"model-serving-api-54d89bd5fc-cngtc\" (UID: \"56d05014-f053-447d-b1d2-37b530a3ef52\") " pod="opendatahub/model-serving-api-54d89bd5fc-cngtc" Jul 24 03:26:34.799806 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:34.798929 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56d05014-f053-447d-b1d2-37b530a3ef52-tls-certs\") pod \"model-serving-api-54d89bd5fc-cngtc\" (UID: \"56d05014-f053-447d-b1d2-37b530a3ef52\") " pod="opendatahub/model-serving-api-54d89bd5fc-cngtc" Jul 24 03:26:34.799806 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:34.799028 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h7z8\" (UniqueName: \"kubernetes.io/projected/56d05014-f053-447d-b1d2-37b530a3ef52-kube-api-access-8h7z8\") pod \"model-serving-api-54d89bd5fc-cngtc\" (UID: \"56d05014-f053-447d-b1d2-37b530a3ef52\") " pod="opendatahub/model-serving-api-54d89bd5fc-cngtc" Jul 24 03:26:34.818541 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:34.818479 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56d05014-f053-447d-b1d2-37b530a3ef52-tls-certs\") pod \"model-serving-api-54d89bd5fc-cngtc\" (UID: \"56d05014-f053-447d-b1d2-37b530a3ef52\") " pod="opendatahub/model-serving-api-54d89bd5fc-cngtc" Jul 24 03:26:34.827846 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:34.827781 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h7z8\" (UniqueName: \"kubernetes.io/projected/56d05014-f053-447d-b1d2-37b530a3ef52-kube-api-access-8h7z8\") pod \"model-serving-api-54d89bd5fc-cngtc\" (UID: \"56d05014-f053-447d-b1d2-37b530a3ef52\") " pod="opendatahub/model-serving-api-54d89bd5fc-cngtc" Jul 24 03:26:34.950101 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:34.950004 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-cngtc" Jul 24 03:26:35.038282 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:35.038239 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:35.041061 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:35.041025 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9cn76" event={"ID":"aa9efd80-0fe0-4b2a-aff1-1a516a4dcc2b","Type":"ContainerStarted","Data":"063dabf7c0812d538fa3fae5b068ba620f9788f48a3f630e5d50b1baf4cf5233"} Jul 24 03:26:35.042993 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:35.042966 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f459d4668-c22g9" event={"ID":"b5b6f8d8-a71c-4088-bc54-52634f3d8eee","Type":"ContainerStarted","Data":"18f118b7dabb4e438ccea30120eb4a4e98c235f3113a1213bccf4294faeae3fe"} Jul 24 03:26:35.153081 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:35.153023 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-cngtc"] Jul 24 03:26:35.158675 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:26:35.158646 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d05014_f053_447d_b1d2_37b530a3ef52.slice/crio-efe0ab2e1a235f036725c0f87d34fb7eeb371a617b887c03b52252b428a6df68 WatchSource:0}: Error finding container efe0ab2e1a235f036725c0f87d34fb7eeb371a617b887c03b52252b428a6df68: Status 404 returned error can't find the container with id efe0ab2e1a235f036725c0f87d34fb7eeb371a617b887c03b52252b428a6df68 Jul 24 03:26:36.008610 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.008516 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:36.018569 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.017071 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct"] Jul 24 03:26:36.020756 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.020725 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" Jul 24 03:26:36.024688 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.024666 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 24 03:26:36.024799 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.024771 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 24 03:26:36.024972 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.024948 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 24 03:26:36.025042 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.024969 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-s5gzl\"" Jul 24 03:26:36.025245 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.025227 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 24 03:26:36.045955 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.045913 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct"] Jul 24 03:26:36.050303 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.050266 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-cngtc" event={"ID":"56d05014-f053-447d-b1d2-37b530a3ef52","Type":"ContainerStarted","Data":"efe0ab2e1a235f036725c0f87d34fb7eeb371a617b887c03b52252b428a6df68"} Jul 24 03:26:36.112488 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.112137 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ee0bf466-30c2-4792-b12d-cd1c46363f54-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-n82ct\" (UID: \"ee0bf466-30c2-4792-b12d-cd1c46363f54\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" Jul 24 03:26:36.112488 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.112212 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee0bf466-30c2-4792-b12d-cd1c46363f54-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-n82ct\" (UID: \"ee0bf466-30c2-4792-b12d-cd1c46363f54\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" Jul 24 03:26:36.112488 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.112247 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0bf466-30c2-4792-b12d-cd1c46363f54-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-n82ct\" (UID: \"ee0bf466-30c2-4792-b12d-cd1c46363f54\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" Jul 24 03:26:36.112488 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.112279 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ee0bf466-30c2-4792-b12d-cd1c46363f54-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-n82ct\" (UID: \"ee0bf466-30c2-4792-b12d-cd1c46363f54\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" Jul 24 03:26:36.112488 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.112307 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ee0bf466-30c2-4792-b12d-cd1c46363f54-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-n82ct\" (UID: \"ee0bf466-30c2-4792-b12d-cd1c46363f54\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" Jul 24 03:26:36.112488 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.112334 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee0bf466-30c2-4792-b12d-cd1c46363f54-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-n82ct\" (UID: \"ee0bf466-30c2-4792-b12d-cd1c46363f54\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" Jul 24 03:26:36.112488 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.112361 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t2m8\" (UniqueName: \"kubernetes.io/projected/ee0bf466-30c2-4792-b12d-cd1c46363f54-kube-api-access-7t2m8\") pod \"istiod-openshift-gateway-7b7995d5cd-n82ct\" (UID: \"ee0bf466-30c2-4792-b12d-cd1c46363f54\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" Jul 24 03:26:36.122086 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.122055 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:36.213479 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.213418 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ee0bf466-30c2-4792-b12d-cd1c46363f54-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-n82ct\" (UID: \"ee0bf466-30c2-4792-b12d-cd1c46363f54\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" Jul 24 03:26:36.213666 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.213511 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee0bf466-30c2-4792-b12d-cd1c46363f54-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-n82ct\" (UID: \"ee0bf466-30c2-4792-b12d-cd1c46363f54\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" Jul 24 03:26:36.213666 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.213548 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0bf466-30c2-4792-b12d-cd1c46363f54-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-n82ct\" (UID: \"ee0bf466-30c2-4792-b12d-cd1c46363f54\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" Jul 24 03:26:36.213666 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.213579 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ee0bf466-30c2-4792-b12d-cd1c46363f54-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-n82ct\" (UID: \"ee0bf466-30c2-4792-b12d-cd1c46363f54\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" Jul 24 03:26:36.213666 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.213607 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ee0bf466-30c2-4792-b12d-cd1c46363f54-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-n82ct\" (UID: \"ee0bf466-30c2-4792-b12d-cd1c46363f54\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" Jul 24 03:26:36.213666 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.213632 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee0bf466-30c2-4792-b12d-cd1c46363f54-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-n82ct\" (UID: \"ee0bf466-30c2-4792-b12d-cd1c46363f54\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" Jul 24 03:26:36.213666 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.213658 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t2m8\" (UniqueName: \"kubernetes.io/projected/ee0bf466-30c2-4792-b12d-cd1c46363f54-kube-api-access-7t2m8\") pod \"istiod-openshift-gateway-7b7995d5cd-n82ct\" (UID: \"ee0bf466-30c2-4792-b12d-cd1c46363f54\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" Jul 24 03:26:36.214183 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.214162 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ee0bf466-30c2-4792-b12d-cd1c46363f54-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-n82ct\" (UID: \"ee0bf466-30c2-4792-b12d-cd1c46363f54\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" Jul 24 03:26:36.217528 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.217194 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ee0bf466-30c2-4792-b12d-cd1c46363f54-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-n82ct\" (UID: \"ee0bf466-30c2-4792-b12d-cd1c46363f54\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" Jul 24 03:26:36.217528 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.217479 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0bf466-30c2-4792-b12d-cd1c46363f54-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-n82ct\" (UID: \"ee0bf466-30c2-4792-b12d-cd1c46363f54\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" Jul 24 03:26:36.217775 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.217560 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ee0bf466-30c2-4792-b12d-cd1c46363f54-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-n82ct\" (UID: \"ee0bf466-30c2-4792-b12d-cd1c46363f54\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" Jul 24 03:26:36.217850 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.217827 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee0bf466-30c2-4792-b12d-cd1c46363f54-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-n82ct\" (UID: \"ee0bf466-30c2-4792-b12d-cd1c46363f54\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" Jul 24 03:26:36.226004 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.225955 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t2m8\" (UniqueName: \"kubernetes.io/projected/ee0bf466-30c2-4792-b12d-cd1c46363f54-kube-api-access-7t2m8\") pod \"istiod-openshift-gateway-7b7995d5cd-n82ct\" (UID: \"ee0bf466-30c2-4792-b12d-cd1c46363f54\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" Jul 24 03:26:36.226321 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.226288 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee0bf466-30c2-4792-b12d-cd1c46363f54-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-n82ct\" (UID: \"ee0bf466-30c2-4792-b12d-cd1c46363f54\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" Jul 24 03:26:36.333654 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:36.333559 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" Jul 24 03:26:37.543824 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:37.543787 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:38.362585 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:38.362559 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct"] Jul 24 03:26:38.364144 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:26:38.364109 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee0bf466_30c2_4792_b12d_cd1c46363f54.slice/crio-7242355eedb97b39e31e39a4e34e68dc9a586f06acdae66f510d01311c3a1910 WatchSource:0}: Error finding container 7242355eedb97b39e31e39a4e34e68dc9a586f06acdae66f510d01311c3a1910: Status 404 returned error can't find the container with id 7242355eedb97b39e31e39a4e34e68dc9a586f06acdae66f510d01311c3a1910 Jul 24 03:26:38.636531 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:38.636455 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:39.065869 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:39.065777 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-cngtc" event={"ID":"56d05014-f053-447d-b1d2-37b530a3ef52","Type":"ContainerStarted","Data":"8220203bfadfb145a867ba666ab28d1bce1ed7dee4131518d4371edcb55eb2c2"} Jul 24 03:26:39.066021 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:39.065928 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-cngtc" Jul 24 03:26:39.067478 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:39.067436 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f459d4668-c22g9" event={"ID":"b5b6f8d8-a71c-4088-bc54-52634f3d8eee","Type":"ContainerStarted","Data":"48836d853ed45f4749e5e126ac4be6020b94a81d1cab9c7720b294f332ab63b7"} Jul 24 03:26:39.068926 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:39.068901 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9cn76" event={"ID":"aa9efd80-0fe0-4b2a-aff1-1a516a4dcc2b","Type":"ContainerStarted","Data":"086fd24c5220309c5461203c56968ec00274888b5b985703b7642bb388f40b43"} Jul 24 03:26:39.069095 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:39.069073 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-9cn76" Jul 24 03:26:39.070248 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:39.070228 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" event={"ID":"ee0bf466-30c2-4792-b12d-cd1c46363f54","Type":"ContainerStarted","Data":"7242355eedb97b39e31e39a4e34e68dc9a586f06acdae66f510d01311c3a1910"} Jul 24 03:26:39.107837 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:39.107737 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-9cn76" podStartSLOduration=2.090517171 podStartE2EDuration="6.107715516s" totalImagesPullingTime="4.017198345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:26:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:26:39.107099325 +0000 UTC m=+548.546518785" watchObservedRunningTime="2026-07-24 03:26:39.107715516 +0000 UTC m=+548.547134977" Jul 24 03:26:39.108348 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:39.108304 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-cngtc" podStartSLOduration=2.074063284 podStartE2EDuration="5.108289274s" totalImagesPullingTime="3.03422599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:26:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:26:39.089948851 +0000 UTC m=+548.529368311" watchObservedRunningTime="2026-07-24 03:26:39.108289274 +0000 UTC m=+548.547708735" Jul 24 03:26:39.131223 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:39.131178 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5f459d4668-c22g9" podStartSLOduration=2.959303744 podStartE2EDuration="7.131163862s" totalImagesPullingTime="4.171860118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:26:39.129560309 +0000 UTC m=+548.568979838" watchObservedRunningTime="2026-07-24 03:26:39.131163862 +0000 UTC m=+548.570583322" Jul 24 03:26:40.982284 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:40.982240 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:26:40.982796 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:40.982772 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:26:41.081615 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:41.081575 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" event={"ID":"ee0bf466-30c2-4792-b12d-cd1c46363f54","Type":"ContainerStarted","Data":"9fa22eb4b9de62d4d4586b1e0ed687bd32c13164e3b0ff322f9dd2f052435ae3"} Jul 24 03:26:41.081780 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:41.081692 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" Jul 24 03:26:41.113877 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:41.113813 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" podStartSLOduration=3.497916353 podStartE2EDuration="6.113791107s" totalImagesPullingTime="2.615874754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:26:41.110839686 +0000 UTC m=+550.550259199" watchObservedRunningTime="2026-07-24 03:26:41.113791107 +0000 UTC m=+550.553210568" Jul 24 03:26:41.576926 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:41.576876 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:41.690592 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:41.690554 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:42.088579 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:42.088531 2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-n82ct container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 03:26:42.088950 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:42.088602 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" podUID="ee0bf466-30c2-4792-b12d-cd1c46363f54" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 03:26:43.036846 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.036817 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp"] Jul 24 03:26:43.040154 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.040127 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.049369 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.049348 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-s2bfb\"" Jul 24 03:26:43.078708 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.078676 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp"] Jul 24 03:26:43.119597 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.119559 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:43.177837 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.177802 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6265fa7d-16b0-46c6-868c-2200a454e4b3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp\" (UID: \"6265fa7d-16b0-46c6-868c-2200a454e4b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.178025 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.177857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6265fa7d-16b0-46c6-868c-2200a454e4b3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp\" (UID: \"6265fa7d-16b0-46c6-868c-2200a454e4b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.178025 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.177946 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6265fa7d-16b0-46c6-868c-2200a454e4b3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp\" (UID: \"6265fa7d-16b0-46c6-868c-2200a454e4b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.178025 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.177984 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6265fa7d-16b0-46c6-868c-2200a454e4b3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp\" (UID: \"6265fa7d-16b0-46c6-868c-2200a454e4b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.178186 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.178029 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6265fa7d-16b0-46c6-868c-2200a454e4b3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp\" (UID: \"6265fa7d-16b0-46c6-868c-2200a454e4b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.178186 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.178051 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6265fa7d-16b0-46c6-868c-2200a454e4b3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp\" (UID: \"6265fa7d-16b0-46c6-868c-2200a454e4b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.178186 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.178100 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6265fa7d-16b0-46c6-868c-2200a454e4b3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp\" (UID: \"6265fa7d-16b0-46c6-868c-2200a454e4b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.178186 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.178135 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jzv5\" (UniqueName: \"kubernetes.io/projected/6265fa7d-16b0-46c6-868c-2200a454e4b3-kube-api-access-9jzv5\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp\" (UID: \"6265fa7d-16b0-46c6-868c-2200a454e4b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.178383 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.178292 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6265fa7d-16b0-46c6-868c-2200a454e4b3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp\" (UID: \"6265fa7d-16b0-46c6-868c-2200a454e4b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.279558 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.279531 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6265fa7d-16b0-46c6-868c-2200a454e4b3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp\" (UID: \"6265fa7d-16b0-46c6-868c-2200a454e4b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.279558 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.279562 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6265fa7d-16b0-46c6-868c-2200a454e4b3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp\" (UID: \"6265fa7d-16b0-46c6-868c-2200a454e4b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.279791 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.279586 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6265fa7d-16b0-46c6-868c-2200a454e4b3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp\" (UID: \"6265fa7d-16b0-46c6-868c-2200a454e4b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.279791 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.279604 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6265fa7d-16b0-46c6-868c-2200a454e4b3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp\" (UID: \"6265fa7d-16b0-46c6-868c-2200a454e4b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.279791 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.279642 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6265fa7d-16b0-46c6-868c-2200a454e4b3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp\" (UID: \"6265fa7d-16b0-46c6-868c-2200a454e4b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.279791 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.279670 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jzv5\" (UniqueName: \"kubernetes.io/projected/6265fa7d-16b0-46c6-868c-2200a454e4b3-kube-api-access-9jzv5\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp\" (UID: \"6265fa7d-16b0-46c6-868c-2200a454e4b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.279791 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.279706 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6265fa7d-16b0-46c6-868c-2200a454e4b3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp\" (UID: \"6265fa7d-16b0-46c6-868c-2200a454e4b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.279791 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.279729 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6265fa7d-16b0-46c6-868c-2200a454e4b3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp\" (UID: \"6265fa7d-16b0-46c6-868c-2200a454e4b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.279791 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.279768 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6265fa7d-16b0-46c6-868c-2200a454e4b3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp\" (UID: \"6265fa7d-16b0-46c6-868c-2200a454e4b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.280126 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.279990 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6265fa7d-16b0-46c6-868c-2200a454e4b3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp\" (UID: \"6265fa7d-16b0-46c6-868c-2200a454e4b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.280126 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.280077 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6265fa7d-16b0-46c6-868c-2200a454e4b3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp\" (UID: \"6265fa7d-16b0-46c6-868c-2200a454e4b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.280203 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.280123 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6265fa7d-16b0-46c6-868c-2200a454e4b3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp\" (UID: \"6265fa7d-16b0-46c6-868c-2200a454e4b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.280234 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.280213 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6265fa7d-16b0-46c6-868c-2200a454e4b3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp\" (UID: \"6265fa7d-16b0-46c6-868c-2200a454e4b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.280329 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.280304 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6265fa7d-16b0-46c6-868c-2200a454e4b3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp\" (UID: \"6265fa7d-16b0-46c6-868c-2200a454e4b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.282185 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.282157 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6265fa7d-16b0-46c6-868c-2200a454e4b3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp\" (UID: \"6265fa7d-16b0-46c6-868c-2200a454e4b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.282267 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.282197 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6265fa7d-16b0-46c6-868c-2200a454e4b3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp\" (UID: \"6265fa7d-16b0-46c6-868c-2200a454e4b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.308563 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.308502 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6265fa7d-16b0-46c6-868c-2200a454e4b3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp\" (UID: \"6265fa7d-16b0-46c6-868c-2200a454e4b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.310249 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.310225 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jzv5\" (UniqueName: \"kubernetes.io/projected/6265fa7d-16b0-46c6-868c-2200a454e4b3-kube-api-access-9jzv5\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp\" (UID: \"6265fa7d-16b0-46c6-868c-2200a454e4b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.350962 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.350939 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:43.386689 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.386661 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:43.501374 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.501344 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:43.522338 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:43.522313 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp"] Jul 24 03:26:43.526011 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:26:43.525986 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6265fa7d_16b0_46c6_868c_2200a454e4b3.slice/crio-5ba333462c074309a1bf7caf555c4ea51936c19b24d0395f2b51951cf08e540d WatchSource:0}: Error finding container 5ba333462c074309a1bf7caf555c4ea51936c19b24d0395f2b51951cf08e540d: Status 404 returned error can't find the container with id 5ba333462c074309a1bf7caf555c4ea51936c19b24d0395f2b51951cf08e540d Jul 24 03:26:44.096310 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:44.096272 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" event={"ID":"6265fa7d-16b0-46c6-868c-2200a454e4b3","Type":"ContainerStarted","Data":"5ba333462c074309a1bf7caf555c4ea51936c19b24d0395f2b51951cf08e540d"} Jul 24 03:26:45.088528 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:45.088496 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" Jul 24 03:26:45.571077 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:45.571042 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:45.834558 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:45.834447 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-dsqpq"] Jul 24 03:26:45.886120 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:45.886054 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:45.916194 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:45.916158 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zhwkc"] Jul 24 03:26:45.916366 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:45.916325 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dsqpq" Jul 24 03:26:45.923841 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:45.923815 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-xglln\"" Jul 24 03:26:45.933614 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:45.933582 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-dsqpq"] Jul 24 03:26:45.933733 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:45.933626 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zhwkc"] Jul 24 03:26:45.933733 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:45.933693 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zhwkc" Jul 24 03:26:45.936745 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:45.936723 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 24 03:26:45.937970 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:45.937950 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 24 03:26:45.940161 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:45.938674 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-k4fqf\"" Jul 24 03:26:46.004890 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:46.004851 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72czq\" (UniqueName: \"kubernetes.io/projected/06e7bf12-474c-4a51-be97-44cc726803f2-kube-api-access-72czq\") pod \"kserve-controller-manager-d6f4b65b8-dsqpq\" (UID: \"06e7bf12-474c-4a51-be97-44cc726803f2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dsqpq" Jul 24 03:26:46.005079 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:46.005039 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/06e7bf12-474c-4a51-be97-44cc726803f2-cert\") pod \"kserve-controller-manager-d6f4b65b8-dsqpq\" (UID: \"06e7bf12-474c-4a51-be97-44cc726803f2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dsqpq" Jul 24 03:26:46.070480 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:46.070424 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:46.105622 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:46.105545 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/06e7bf12-474c-4a51-be97-44cc726803f2-cert\") pod \"kserve-controller-manager-d6f4b65b8-dsqpq\" (UID: \"06e7bf12-474c-4a51-be97-44cc726803f2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dsqpq" Jul 24 03:26:46.105622 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:46.105603 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b03c41a-0004-44ea-ab51-ac94dde9c9a0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zhwkc\" (UID: \"2b03c41a-0004-44ea-ab51-ac94dde9c9a0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zhwkc" Jul 24 03:26:46.106138 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:46.105646 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72czq\" (UniqueName: \"kubernetes.io/projected/06e7bf12-474c-4a51-be97-44cc726803f2-kube-api-access-72czq\") pod \"kserve-controller-manager-d6f4b65b8-dsqpq\" (UID: \"06e7bf12-474c-4a51-be97-44cc726803f2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dsqpq" Jul 24 03:26:46.106138 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:46.105707 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgtp2\" (UniqueName: \"kubernetes.io/projected/2b03c41a-0004-44ea-ab51-ac94dde9c9a0-kube-api-access-pgtp2\") pod \"llmisvc-controller-manager-777c6bd7d5-zhwkc\" (UID: \"2b03c41a-0004-44ea-ab51-ac94dde9c9a0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zhwkc" Jul 24 03:26:46.108568 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:46.108542 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/06e7bf12-474c-4a51-be97-44cc726803f2-cert\") pod \"kserve-controller-manager-d6f4b65b8-dsqpq\" (UID: \"06e7bf12-474c-4a51-be97-44cc726803f2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dsqpq" Jul 24 03:26:46.123326 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:46.123290 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72czq\" (UniqueName: \"kubernetes.io/projected/06e7bf12-474c-4a51-be97-44cc726803f2-kube-api-access-72czq\") pod \"kserve-controller-manager-d6f4b65b8-dsqpq\" (UID: \"06e7bf12-474c-4a51-be97-44cc726803f2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dsqpq" Jul 24 03:26:46.207424 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:46.206577 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgtp2\" (UniqueName: \"kubernetes.io/projected/2b03c41a-0004-44ea-ab51-ac94dde9c9a0-kube-api-access-pgtp2\") pod \"llmisvc-controller-manager-777c6bd7d5-zhwkc\" (UID: \"2b03c41a-0004-44ea-ab51-ac94dde9c9a0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zhwkc" Jul 24 03:26:46.207424 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:46.206729 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b03c41a-0004-44ea-ab51-ac94dde9c9a0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zhwkc\" (UID: \"2b03c41a-0004-44ea-ab51-ac94dde9c9a0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zhwkc" Jul 24 03:26:46.210403 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:46.210365 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b03c41a-0004-44ea-ab51-ac94dde9c9a0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-zhwkc\" (UID: \"2b03c41a-0004-44ea-ab51-ac94dde9c9a0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zhwkc" Jul 24 03:26:46.221796 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:46.221755 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgtp2\" (UniqueName: \"kubernetes.io/projected/2b03c41a-0004-44ea-ab51-ac94dde9c9a0-kube-api-access-pgtp2\") pod \"llmisvc-controller-manager-777c6bd7d5-zhwkc\" (UID: \"2b03c41a-0004-44ea-ab51-ac94dde9c9a0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zhwkc" Jul 24 03:26:46.230431 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:46.230406 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dsqpq" Jul 24 03:26:46.239673 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:46.239644 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:46.248278 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:46.248259 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zhwkc" Jul 24 03:26:46.581568 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:46.581535 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-zhwkc"] Jul 24 03:26:46.583234 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:26:46.583198 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2b03c41a_0004_44ea_ab51_ac94dde9c9a0.slice/crio-958732548c161109569143383eeb03707e3ecd0606a1afb602b89b8f0e547739 WatchSource:0}: Error finding container 958732548c161109569143383eeb03707e3ecd0606a1afb602b89b8f0e547739: Status 404 returned error can't find the container with id 958732548c161109569143383eeb03707e3ecd0606a1afb602b89b8f0e547739 Jul 24 03:26:46.609766 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:46.609740 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-dsqpq"] Jul 24 03:26:46.611854 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:26:46.611827 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e7bf12_474c_4a51_be97_44cc726803f2.slice/crio-4808e0d867d818cc1a8308ae61fff3350d33c5405947238b73a9a35fbee51ab7 WatchSource:0}: Error finding container 4808e0d867d818cc1a8308ae61fff3350d33c5405947238b73a9a35fbee51ab7: Status 404 returned error can't find the container with id 4808e0d867d818cc1a8308ae61fff3350d33c5405947238b73a9a35fbee51ab7 Jul 24 03:26:46.620023 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:46.619992 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:26:46.620107 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:46.620053 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:26:46.620107 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:46.620082 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:26:47.114222 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:47.114188 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dsqpq" event={"ID":"06e7bf12-474c-4a51-be97-44cc726803f2","Type":"ContainerStarted","Data":"4808e0d867d818cc1a8308ae61fff3350d33c5405947238b73a9a35fbee51ab7"} Jul 24 03:26:47.116152 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:47.116122 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" event={"ID":"6265fa7d-16b0-46c6-868c-2200a454e4b3","Type":"ContainerStarted","Data":"f07d6094db9d8f5b0e417c249a4e4d064936dfcaefbb338618fd00c678aec861"} Jul 24 03:26:47.117393 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:47.117353 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zhwkc" event={"ID":"2b03c41a-0004-44ea-ab51-ac94dde9c9a0","Type":"ContainerStarted","Data":"958732548c161109569143383eeb03707e3ecd0606a1afb602b89b8f0e547739"} Jul 24 03:26:47.161233 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:47.161182 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" podStartSLOduration=1.069390902 podStartE2EDuration="4.161161981s" totalImagesPullingTime="3.091771079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:26:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:26:47.158315483 +0000 UTC m=+556.597734950" watchObservedRunningTime="2026-07-24 03:26:47.161161981 +0000 UTC m=+556.600581441" Jul 24 03:26:47.351185 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:47.351154 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:47.356788 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:47.356762 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:47.554686 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:47.554597 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:47.696218 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:47.696182 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:47.799726 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:47.799687 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:26:48.122812 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:48.122781 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:48.123897 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:48.123868 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp" Jul 24 03:26:49.089719 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:49.079430 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct"] Jul 24 03:26:49.090388 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:49.090179 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" podUID="ee0bf466-30c2-4792-b12d-cd1c46363f54" containerName="discovery" containerID="cri-o://9fa22eb4b9de62d4d4586b1e0ed687bd32c13164e3b0ff322f9dd2f052435ae3" gracePeriod=30 Jul 24 03:26:49.623161 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:49.623136 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" Jul 24 03:26:49.739763 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:49.739663 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ee0bf466-30c2-4792-b12d-cd1c46363f54-istio-csr-ca-configmap\") pod \"ee0bf466-30c2-4792-b12d-cd1c46363f54\" (UID: \"ee0bf466-30c2-4792-b12d-cd1c46363f54\") " Jul 24 03:26:49.739763 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:49.739737 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ee0bf466-30c2-4792-b12d-cd1c46363f54-cacerts\") pod \"ee0bf466-30c2-4792-b12d-cd1c46363f54\" (UID: \"ee0bf466-30c2-4792-b12d-cd1c46363f54\") " Jul 24 03:26:49.739964 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:49.739850 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0bf466-30c2-4792-b12d-cd1c46363f54-local-certs\") pod \"ee0bf466-30c2-4792-b12d-cd1c46363f54\" (UID: \"ee0bf466-30c2-4792-b12d-cd1c46363f54\") " Jul 24 03:26:49.739964 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:49.739888 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee0bf466-30c2-4792-b12d-cd1c46363f54-istio-kubeconfig\") pod \"ee0bf466-30c2-4792-b12d-cd1c46363f54\" (UID: \"ee0bf466-30c2-4792-b12d-cd1c46363f54\") " Jul 24 03:26:49.739964 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:49.739921 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ee0bf466-30c2-4792-b12d-cd1c46363f54-istio-csr-dns-cert\") pod \"ee0bf466-30c2-4792-b12d-cd1c46363f54\" (UID: \"ee0bf466-30c2-4792-b12d-cd1c46363f54\") " Jul 24 03:26:49.740124 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:49.739975 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7t2m8\" (UniqueName: \"kubernetes.io/projected/ee0bf466-30c2-4792-b12d-cd1c46363f54-kube-api-access-7t2m8\") pod \"ee0bf466-30c2-4792-b12d-cd1c46363f54\" (UID: \"ee0bf466-30c2-4792-b12d-cd1c46363f54\") " Jul 24 03:26:49.740124 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:49.740016 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee0bf466-30c2-4792-b12d-cd1c46363f54-istio-token\") pod \"ee0bf466-30c2-4792-b12d-cd1c46363f54\" (UID: \"ee0bf466-30c2-4792-b12d-cd1c46363f54\") " Jul 24 03:26:49.740124 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:49.740084 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee0bf466-30c2-4792-b12d-cd1c46363f54-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "ee0bf466-30c2-4792-b12d-cd1c46363f54" (UID: "ee0bf466-30c2-4792-b12d-cd1c46363f54"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 03:26:49.740333 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:49.740315 2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ee0bf466-30c2-4792-b12d-cd1c46363f54-istio-csr-ca-configmap\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:26:49.742274 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:49.742243 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0bf466-30c2-4792-b12d-cd1c46363f54-local-certs" (OuterVolumeSpecName: "local-certs") pod "ee0bf466-30c2-4792-b12d-cd1c46363f54" (UID: "ee0bf466-30c2-4792-b12d-cd1c46363f54"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:26:49.742641 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:49.742620 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee0bf466-30c2-4792-b12d-cd1c46363f54-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "ee0bf466-30c2-4792-b12d-cd1c46363f54" (UID: "ee0bf466-30c2-4792-b12d-cd1c46363f54"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 03:26:49.742783 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:49.742751 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee0bf466-30c2-4792-b12d-cd1c46363f54-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "ee0bf466-30c2-4792-b12d-cd1c46363f54" (UID: "ee0bf466-30c2-4792-b12d-cd1c46363f54"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 03:26:49.742783 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:49.742771 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee0bf466-30c2-4792-b12d-cd1c46363f54-cacerts" (OuterVolumeSpecName: "cacerts") pod "ee0bf466-30c2-4792-b12d-cd1c46363f54" (UID: "ee0bf466-30c2-4792-b12d-cd1c46363f54"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 03:26:49.742872 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:49.742780 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee0bf466-30c2-4792-b12d-cd1c46363f54-kube-api-access-7t2m8" (OuterVolumeSpecName: "kube-api-access-7t2m8") pod "ee0bf466-30c2-4792-b12d-cd1c46363f54" (UID: "ee0bf466-30c2-4792-b12d-cd1c46363f54"). InnerVolumeSpecName "kube-api-access-7t2m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:26:49.742872 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:49.742838 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee0bf466-30c2-4792-b12d-cd1c46363f54-istio-token" (OuterVolumeSpecName: "istio-token") pod "ee0bf466-30c2-4792-b12d-cd1c46363f54" (UID: "ee0bf466-30c2-4792-b12d-cd1c46363f54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:26:49.841113 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:49.841081 2579 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee0bf466-30c2-4792-b12d-cd1c46363f54-istio-kubeconfig\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:26:49.841113 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:49.841108 2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ee0bf466-30c2-4792-b12d-cd1c46363f54-istio-csr-dns-cert\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:26:49.841113 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:49.841117 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t2m8\" (UniqueName: \"kubernetes.io/projected/ee0bf466-30c2-4792-b12d-cd1c46363f54-kube-api-access-7t2m8\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:26:49.841323 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:49.841126 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee0bf466-30c2-4792-b12d-cd1c46363f54-istio-token\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:26:49.841323 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:49.841137 2579 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ee0bf466-30c2-4792-b12d-cd1c46363f54-cacerts\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:26:49.841323 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:49.841145 2579 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee0bf466-30c2-4792-b12d-cd1c46363f54-local-certs\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:26:50.078671 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:50.078586 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-9cn76" Jul 24 03:26:50.080528 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:50.080512 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-cngtc" Jul 24 03:26:50.134897 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:50.134867 2579 generic.go:356] "Generic (PLEG): container finished" podID="ee0bf466-30c2-4792-b12d-cd1c46363f54" containerID="9fa22eb4b9de62d4d4586b1e0ed687bd32c13164e3b0ff322f9dd2f052435ae3" exitCode=0 Jul 24 03:26:50.135056 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:50.134917 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" Jul 24 03:26:50.135056 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:50.134949 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" event={"ID":"ee0bf466-30c2-4792-b12d-cd1c46363f54","Type":"ContainerDied","Data":"9fa22eb4b9de62d4d4586b1e0ed687bd32c13164e3b0ff322f9dd2f052435ae3"} Jul 24 03:26:50.135056 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:50.134995 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct" event={"ID":"ee0bf466-30c2-4792-b12d-cd1c46363f54","Type":"ContainerDied","Data":"7242355eedb97b39e31e39a4e34e68dc9a586f06acdae66f510d01311c3a1910"} Jul 24 03:26:50.135056 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:50.135030 2579 scope.go:117] "RemoveContainer" containerID="9fa22eb4b9de62d4d4586b1e0ed687bd32c13164e3b0ff322f9dd2f052435ae3" Jul 24 03:26:50.139000 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:50.138961 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zhwkc" event={"ID":"2b03c41a-0004-44ea-ab51-ac94dde9c9a0","Type":"ContainerStarted","Data":"c900613f02fdefbe8121c25bcd7ac5a798ba06f8f3737b0227bbd4f34299f156"} Jul 24 03:26:50.143075 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:50.139806 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zhwkc" Jul 24 03:26:50.146877 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:50.146847 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dsqpq" event={"ID":"06e7bf12-474c-4a51-be97-44cc726803f2","Type":"ContainerStarted","Data":"87cbd5483f66827d5d5db0f122dc238fd5be39874e4f057f968b1ccfda6d018a"} Jul 24 03:26:50.147101 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:50.147082 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dsqpq" Jul 24 03:26:50.151973 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:50.151956 2579 scope.go:117] "RemoveContainer" containerID="9fa22eb4b9de62d4d4586b1e0ed687bd32c13164e3b0ff322f9dd2f052435ae3" Jul 24 03:26:50.152344 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:26:50.152316 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fa22eb4b9de62d4d4586b1e0ed687bd32c13164e3b0ff322f9dd2f052435ae3\": container with ID starting with 9fa22eb4b9de62d4d4586b1e0ed687bd32c13164e3b0ff322f9dd2f052435ae3 not found: ID does not exist" containerID="9fa22eb4b9de62d4d4586b1e0ed687bd32c13164e3b0ff322f9dd2f052435ae3" Jul 24 03:26:50.152435 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:50.152352 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fa22eb4b9de62d4d4586b1e0ed687bd32c13164e3b0ff322f9dd2f052435ae3"} err="failed to get container status \"9fa22eb4b9de62d4d4586b1e0ed687bd32c13164e3b0ff322f9dd2f052435ae3\": rpc error: code = NotFound desc = could not find container \"9fa22eb4b9de62d4d4586b1e0ed687bd32c13164e3b0ff322f9dd2f052435ae3\": container with ID starting with 9fa22eb4b9de62d4d4586b1e0ed687bd32c13164e3b0ff322f9dd2f052435ae3 not found: ID does not exist" Jul 24 03:26:50.196595 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:50.196538 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zhwkc" podStartSLOduration=2.284495046 podStartE2EDuration="5.196520579s" totalImagesPullingTime="2.912025533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:26:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:26:50.191249658 +0000 UTC m=+559.630669118" watchObservedRunningTime="2026-07-24 03:26:50.196520579 +0000 UTC m=+559.635940017" Jul 24 03:26:50.215947 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:50.215895 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dsqpq" podStartSLOduration=2.3362947370000002 podStartE2EDuration="5.215879482s" totalImagesPullingTime="2.879584745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:26:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:26:50.213887883 +0000 UTC m=+559.653307343" watchObservedRunningTime="2026-07-24 03:26:50.215879482 +0000 UTC m=+559.655298942" Jul 24 03:26:50.228635 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:50.228607 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct"] Jul 24 03:26:50.232722 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:50.232700 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n82ct"] Jul 24 03:26:51.033075 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:26:51.033037 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee0bf466-30c2-4792-b12d-cd1c46363f54" path="/var/lib/kubelet/pods/ee0bf466-30c2-4792-b12d-cd1c46363f54/volumes" Jul 24 03:27:22.158954 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:22.158925 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-zhwkc" Jul 24 03:27:22.161955 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:22.161937 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dsqpq" Jul 24 03:27:29.665762 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:29.665716 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:27:30.951776 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:30.951742 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:27:33.202339 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:33.202309 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:27:33.302839 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:33.302804 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:27:33.335080 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:33.335042 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bsqsq"] Jul 24 03:27:33.335406 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:33.335394 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee0bf466-30c2-4792-b12d-cd1c46363f54" containerName="discovery" Jul 24 03:27:33.335487 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:33.335408 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee0bf466-30c2-4792-b12d-cd1c46363f54" containerName="discovery" Jul 24 03:27:33.335487 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:33.335484 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee0bf466-30c2-4792-b12d-cd1c46363f54" containerName="discovery" Jul 24 03:27:33.338311 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:33.338290 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bsqsq" Jul 24 03:27:33.341206 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:33.341188 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 03:27:33.341909 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:33.341894 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 03:27:33.341987 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:33.341905 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-79spw\"" Jul 24 03:27:33.351816 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:33.351788 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bsqsq"] Jul 24 03:27:33.393987 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:33.393961 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:27:33.423258 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:33.423229 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbzdl\" (UniqueName: \"kubernetes.io/projected/23bc1d2e-cf7e-4cb6-92f0-e60df02313b2-kube-api-access-zbzdl\") pod \"kuadrant-operator-catalog-bsqsq\" (UID: \"23bc1d2e-cf7e-4cb6-92f0-e60df02313b2\") " pod="kuadrant-system/kuadrant-operator-catalog-bsqsq" Jul 24 03:27:33.523893 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:33.523794 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbzdl\" (UniqueName: \"kubernetes.io/projected/23bc1d2e-cf7e-4cb6-92f0-e60df02313b2-kube-api-access-zbzdl\") pod \"kuadrant-operator-catalog-bsqsq\" (UID: \"23bc1d2e-cf7e-4cb6-92f0-e60df02313b2\") " pod="kuadrant-system/kuadrant-operator-catalog-bsqsq" Jul 24 03:27:33.531527 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:33.531506 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbzdl\" (UniqueName: \"kubernetes.io/projected/23bc1d2e-cf7e-4cb6-92f0-e60df02313b2-kube-api-access-zbzdl\") pod \"kuadrant-operator-catalog-bsqsq\" (UID: \"23bc1d2e-cf7e-4cb6-92f0-e60df02313b2\") " pod="kuadrant-system/kuadrant-operator-catalog-bsqsq" Jul 24 03:27:33.648684 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:33.648650 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bsqsq" Jul 24 03:27:33.721120 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:33.721077 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bsqsq"] Jul 24 03:27:33.788218 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:33.788143 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bsqsq"] Jul 24 03:27:33.792300 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:27:33.792274 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23bc1d2e_cf7e_4cb6_92f0_e60df02313b2.slice/crio-8c82533cb4e4a2d7e67be288d9be0946d32e08d047e2684c2b2e71af665ee0dc WatchSource:0}: Error finding container 8c82533cb4e4a2d7e67be288d9be0946d32e08d047e2684c2b2e71af665ee0dc: Status 404 returned error can't find the container with id 8c82533cb4e4a2d7e67be288d9be0946d32e08d047e2684c2b2e71af665ee0dc Jul 24 03:27:33.924031 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:33.923974 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ttwx2"] Jul 24 03:27:33.929914 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:33.929883 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ttwx2" Jul 24 03:27:33.938870 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:33.938841 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ttwx2"] Jul 24 03:27:34.027646 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:34.027609 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn8mx\" (UniqueName: \"kubernetes.io/projected/3e799fcf-1581-48ce-94c2-f8c977bf6d42-kube-api-access-jn8mx\") pod \"kuadrant-operator-catalog-ttwx2\" (UID: \"3e799fcf-1581-48ce-94c2-f8c977bf6d42\") " pod="kuadrant-system/kuadrant-operator-catalog-ttwx2" Jul 24 03:27:34.128376 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:34.128345 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn8mx\" (UniqueName: \"kubernetes.io/projected/3e799fcf-1581-48ce-94c2-f8c977bf6d42-kube-api-access-jn8mx\") pod \"kuadrant-operator-catalog-ttwx2\" (UID: \"3e799fcf-1581-48ce-94c2-f8c977bf6d42\") " pod="kuadrant-system/kuadrant-operator-catalog-ttwx2" Jul 24 03:27:34.138895 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:34.138865 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn8mx\" (UniqueName: \"kubernetes.io/projected/3e799fcf-1581-48ce-94c2-f8c977bf6d42-kube-api-access-jn8mx\") pod \"kuadrant-operator-catalog-ttwx2\" (UID: \"3e799fcf-1581-48ce-94c2-f8c977bf6d42\") " pod="kuadrant-system/kuadrant-operator-catalog-ttwx2" Jul 24 03:27:34.161627 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:34.161597 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:27:34.240793 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:34.240757 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ttwx2" Jul 24 03:27:34.340429 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:34.340393 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bsqsq" event={"ID":"23bc1d2e-cf7e-4cb6-92f0-e60df02313b2","Type":"ContainerStarted","Data":"8c82533cb4e4a2d7e67be288d9be0946d32e08d047e2684c2b2e71af665ee0dc"} Jul 24 03:27:34.363117 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:34.363080 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:27:34.386609 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:34.386387 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ttwx2"] Jul 24 03:27:34.431282 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:27:34.431249 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e799fcf_1581_48ce_94c2_f8c977bf6d42.slice/crio-9ac4133306351d2deea562ffd2e15d174847b88ef96c485b0cf06922de754bb2 WatchSource:0}: Error finding container 9ac4133306351d2deea562ffd2e15d174847b88ef96c485b0cf06922de754bb2: Status 404 returned error can't find the container with id 9ac4133306351d2deea562ffd2e15d174847b88ef96c485b0cf06922de754bb2 Jul 24 03:27:35.347630 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:35.347601 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ttwx2" event={"ID":"3e799fcf-1581-48ce-94c2-f8c977bf6d42","Type":"ContainerStarted","Data":"9ac4133306351d2deea562ffd2e15d174847b88ef96c485b0cf06922de754bb2"} Jul 24 03:27:36.353319 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:36.353279 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bsqsq" event={"ID":"23bc1d2e-cf7e-4cb6-92f0-e60df02313b2","Type":"ContainerStarted","Data":"74acb1d7cb5c8bd4544fd7f8e57ca918de871d79fc2c5a530ffe3786c362a239"} Jul 24 03:27:36.353319 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:36.353320 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-bsqsq" podUID="23bc1d2e-cf7e-4cb6-92f0-e60df02313b2" containerName="registry-server" containerID="cri-o://74acb1d7cb5c8bd4544fd7f8e57ca918de871d79fc2c5a530ffe3786c362a239" gracePeriod=2 Jul 24 03:27:36.355012 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:36.354983 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ttwx2" event={"ID":"3e799fcf-1581-48ce-94c2-f8c977bf6d42","Type":"ContainerStarted","Data":"af34a7b8099bd929806ac39b723611b5a941c4bd2735733ad8efd4b403a6e7e4"} Jul 24 03:27:36.368518 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:36.368452 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-bsqsq" podStartSLOduration=1.389055245 podStartE2EDuration="3.368435704s" totalImagesPullingTime="1.979380459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:27:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:27:36.366997903 +0000 UTC m=+605.806417364" watchObservedRunningTime="2026-07-24 03:27:36.368435704 +0000 UTC m=+605.807855195" Jul 24 03:27:36.383020 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:36.382971 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-ttwx2" podStartSLOduration=2.041929077 podStartE2EDuration="3.382948022s" totalImagesPullingTime="1.341018945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:27:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:27:36.380926245 +0000 UTC m=+605.820345718" watchObservedRunningTime="2026-07-24 03:27:36.382948022 +0000 UTC m=+605.822367478" Jul 24 03:27:36.603918 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:36.603860 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bsqsq" Jul 24 03:27:36.648050 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:36.648015 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbzdl\" (UniqueName: \"kubernetes.io/projected/23bc1d2e-cf7e-4cb6-92f0-e60df02313b2-kube-api-access-zbzdl\") pod \"23bc1d2e-cf7e-4cb6-92f0-e60df02313b2\" (UID: \"23bc1d2e-cf7e-4cb6-92f0-e60df02313b2\") " Jul 24 03:27:36.650235 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:36.650213 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23bc1d2e-cf7e-4cb6-92f0-e60df02313b2-kube-api-access-zbzdl" (OuterVolumeSpecName: "kube-api-access-zbzdl") pod "23bc1d2e-cf7e-4cb6-92f0-e60df02313b2" (UID: "23bc1d2e-cf7e-4cb6-92f0-e60df02313b2"). InnerVolumeSpecName "kube-api-access-zbzdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:27:36.748895 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:36.748862 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbzdl\" (UniqueName: \"kubernetes.io/projected/23bc1d2e-cf7e-4cb6-92f0-e60df02313b2-kube-api-access-zbzdl\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:27:37.360430 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:37.360397 2579 generic.go:356] "Generic (PLEG): container finished" podID="23bc1d2e-cf7e-4cb6-92f0-e60df02313b2" containerID="74acb1d7cb5c8bd4544fd7f8e57ca918de871d79fc2c5a530ffe3786c362a239" exitCode=0 Jul 24 03:27:37.360876 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:37.360445 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bsqsq" Jul 24 03:27:37.360876 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:37.360488 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bsqsq" event={"ID":"23bc1d2e-cf7e-4cb6-92f0-e60df02313b2","Type":"ContainerDied","Data":"74acb1d7cb5c8bd4544fd7f8e57ca918de871d79fc2c5a530ffe3786c362a239"} Jul 24 03:27:37.360876 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:37.360545 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bsqsq" event={"ID":"23bc1d2e-cf7e-4cb6-92f0-e60df02313b2","Type":"ContainerDied","Data":"8c82533cb4e4a2d7e67be288d9be0946d32e08d047e2684c2b2e71af665ee0dc"} Jul 24 03:27:37.360876 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:37.360571 2579 scope.go:117] "RemoveContainer" containerID="74acb1d7cb5c8bd4544fd7f8e57ca918de871d79fc2c5a530ffe3786c362a239" Jul 24 03:27:37.369572 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:37.369552 2579 scope.go:117] "RemoveContainer" containerID="74acb1d7cb5c8bd4544fd7f8e57ca918de871d79fc2c5a530ffe3786c362a239" Jul 24 03:27:37.369815 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:27:37.369795 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74acb1d7cb5c8bd4544fd7f8e57ca918de871d79fc2c5a530ffe3786c362a239\": container with ID starting with 74acb1d7cb5c8bd4544fd7f8e57ca918de871d79fc2c5a530ffe3786c362a239 not found: ID does not exist" containerID="74acb1d7cb5c8bd4544fd7f8e57ca918de871d79fc2c5a530ffe3786c362a239" Jul 24 03:27:37.369867 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:37.369824 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74acb1d7cb5c8bd4544fd7f8e57ca918de871d79fc2c5a530ffe3786c362a239"} err="failed to get container status \"74acb1d7cb5c8bd4544fd7f8e57ca918de871d79fc2c5a530ffe3786c362a239\": rpc error: code = NotFound desc = could not find container \"74acb1d7cb5c8bd4544fd7f8e57ca918de871d79fc2c5a530ffe3786c362a239\": container with ID starting with 74acb1d7cb5c8bd4544fd7f8e57ca918de871d79fc2c5a530ffe3786c362a239 not found: ID does not exist" Jul 24 03:27:37.375782 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:37.375762 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bsqsq"] Jul 24 03:27:37.378262 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:37.378243 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bsqsq"] Jul 24 03:27:39.033724 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:39.033696 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23bc1d2e-cf7e-4cb6-92f0-e60df02313b2" path="/var/lib/kubelet/pods/23bc1d2e-cf7e-4cb6-92f0-e60df02313b2/volumes" Jul 24 03:27:44.241043 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:44.241007 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-ttwx2" Jul 24 03:27:44.241525 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:44.241081 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-ttwx2" Jul 24 03:27:44.262807 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:44.262784 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-ttwx2" Jul 24 03:27:44.412762 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:27:44.412733 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-ttwx2" Jul 24 03:28:02.628197 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:02.628096 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:05.203427 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:05.203390 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-twsqn"] Jul 24 03:28:05.203812 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:05.203726 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23bc1d2e-cf7e-4cb6-92f0-e60df02313b2" containerName="registry-server" Jul 24 03:28:05.203812 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:05.203737 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="23bc1d2e-cf7e-4cb6-92f0-e60df02313b2" containerName="registry-server" Jul 24 03:28:05.203812 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:05.203789 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="23bc1d2e-cf7e-4cb6-92f0-e60df02313b2" containerName="registry-server" Jul 24 03:28:05.206703 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:05.206688 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-twsqn" Jul 24 03:28:05.209146 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:05.209122 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 24 03:28:05.209267 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:05.209166 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-7wpsn\"" Jul 24 03:28:05.219141 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:05.219101 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-twsqn"] Jul 24 03:28:05.261357 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:05.261329 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47jkp\" (UniqueName: \"kubernetes.io/projected/7c5ca5e8-c8b2-4f7c-a7e8-43594cc7635a-kube-api-access-47jkp\") pod \"dns-operator-controller-manager-864dd78c4c-twsqn\" (UID: \"7c5ca5e8-c8b2-4f7c-a7e8-43594cc7635a\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-twsqn" Jul 24 03:28:05.362370 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:05.362332 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47jkp\" (UniqueName: \"kubernetes.io/projected/7c5ca5e8-c8b2-4f7c-a7e8-43594cc7635a-kube-api-access-47jkp\") pod \"dns-operator-controller-manager-864dd78c4c-twsqn\" (UID: \"7c5ca5e8-c8b2-4f7c-a7e8-43594cc7635a\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-twsqn" Jul 24 03:28:05.370426 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:05.370399 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47jkp\" (UniqueName: \"kubernetes.io/projected/7c5ca5e8-c8b2-4f7c-a7e8-43594cc7635a-kube-api-access-47jkp\") pod \"dns-operator-controller-manager-864dd78c4c-twsqn\" (UID: \"7c5ca5e8-c8b2-4f7c-a7e8-43594cc7635a\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-twsqn" Jul 24 03:28:05.517340 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:05.517257 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-twsqn" Jul 24 03:28:05.649980 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:05.649946 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-twsqn"] Jul 24 03:28:05.652887 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:28:05.652850 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c5ca5e8_c8b2_4f7c_a7e8_43594cc7635a.slice/crio-0c6af07af6d1fae8a216f6dc16f48085beda0d8197aebd45c5d501817bab92d6 WatchSource:0}: Error finding container 0c6af07af6d1fae8a216f6dc16f48085beda0d8197aebd45c5d501817bab92d6: Status 404 returned error can't find the container with id 0c6af07af6d1fae8a216f6dc16f48085beda0d8197aebd45c5d501817bab92d6 Jul 24 03:28:06.488101 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:06.488062 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-twsqn" event={"ID":"7c5ca5e8-c8b2-4f7c-a7e8-43594cc7635a","Type":"ContainerStarted","Data":"0c6af07af6d1fae8a216f6dc16f48085beda0d8197aebd45c5d501817bab92d6"} Jul 24 03:28:06.596090 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:06.596053 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:06.697965 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:06.697930 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:06.797155 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:06.797071 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:07.493536 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:07.493505 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-twsqn" event={"ID":"7c5ca5e8-c8b2-4f7c-a7e8-43594cc7635a","Type":"ContainerStarted","Data":"ad15691aa2183272feed4c35be69d2b676def0c71aff03c11976ed4ab74250db"} Jul 24 03:28:07.493946 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:07.493592 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-twsqn" Jul 24 03:28:07.510645 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:07.510596 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-twsqn" podStartSLOduration=0.795136397 podStartE2EDuration="2.510581305s" totalImagesPullingTime="1.715444908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:28:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:28:07.508714905 +0000 UTC m=+636.948134362" watchObservedRunningTime="2026-07-24 03:28:07.510581305 +0000 UTC m=+636.950000765" Jul 24 03:28:08.669765 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:08.669732 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8g9dl"] Jul 24 03:28:08.673285 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:08.673266 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8g9dl" Jul 24 03:28:08.675930 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:08.675910 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-lntk2\"" Jul 24 03:28:08.683867 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:08.683847 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8g9dl"] Jul 24 03:28:08.792756 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:08.792725 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzkvb\" (UniqueName: \"kubernetes.io/projected/06c5c9db-7be6-49e3-850d-ddc5658c992c-kube-api-access-nzkvb\") pod \"limitador-operator-controller-manager-859d7b78bd-8g9dl\" (UID: \"06c5c9db-7be6-49e3-850d-ddc5658c992c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8g9dl" Jul 24 03:28:08.893567 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:08.893534 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzkvb\" (UniqueName: \"kubernetes.io/projected/06c5c9db-7be6-49e3-850d-ddc5658c992c-kube-api-access-nzkvb\") pod \"limitador-operator-controller-manager-859d7b78bd-8g9dl\" (UID: \"06c5c9db-7be6-49e3-850d-ddc5658c992c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8g9dl" Jul 24 03:28:08.901678 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:08.901649 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzkvb\" (UniqueName: \"kubernetes.io/projected/06c5c9db-7be6-49e3-850d-ddc5658c992c-kube-api-access-nzkvb\") pod \"limitador-operator-controller-manager-859d7b78bd-8g9dl\" (UID: \"06c5c9db-7be6-49e3-850d-ddc5658c992c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8g9dl" Jul 24 03:28:08.984180 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:08.984091 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8g9dl" Jul 24 03:28:09.115688 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:09.115554 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8g9dl"] Jul 24 03:28:09.117879 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:28:09.117855 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c5c9db_7be6_49e3_850d_ddc5658c992c.slice/crio-830abb86b40d993a45a8a4e4664e084d18b7805b6eda8b154635e6a19cf083d7 WatchSource:0}: Error finding container 830abb86b40d993a45a8a4e4664e084d18b7805b6eda8b154635e6a19cf083d7: Status 404 returned error can't find the container with id 830abb86b40d993a45a8a4e4664e084d18b7805b6eda8b154635e6a19cf083d7 Jul 24 03:28:09.234430 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:09.234345 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:09.388140 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:09.388108 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:09.502523 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:09.502441 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8g9dl" event={"ID":"06c5c9db-7be6-49e3-850d-ddc5658c992c","Type":"ContainerStarted","Data":"830abb86b40d993a45a8a4e4664e084d18b7805b6eda8b154635e6a19cf083d7"} Jul 24 03:28:10.412657 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:10.412616 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:11.513260 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:11.513215 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8g9dl" event={"ID":"06c5c9db-7be6-49e3-850d-ddc5658c992c","Type":"ContainerStarted","Data":"d8e7671fc6464a22b2ab48d6c85a0fcaea901f1b2e7990b68429463720fab6a9"} Jul 24 03:28:11.513728 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:11.513280 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8g9dl" Jul 24 03:28:11.535814 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:11.531942 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8g9dl" podStartSLOduration=2.088250353 podStartE2EDuration="3.531924629s" totalImagesPullingTime="1.443674276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:28:11.530421156 +0000 UTC m=+640.969840616" watchObservedRunningTime="2026-07-24 03:28:11.531924629 +0000 UTC m=+640.971344091" Jul 24 03:28:13.893975 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:13.893938 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht"] Jul 24 03:28:13.897828 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:13.897806 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht" Jul 24 03:28:13.900226 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:13.900207 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-44f58\"" Jul 24 03:28:13.914747 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:13.914726 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht"] Jul 24 03:28:13.934703 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:13.934674 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trcls\" (UniqueName: \"kubernetes.io/projected/29c111eb-b99b-4c47-94ec-373c4f1e656a-kube-api-access-trcls\") pod \"kuadrant-operator-controller-manager-8684c9ff49-s2vht\" (UID: \"29c111eb-b99b-4c47-94ec-373c4f1e656a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht" Jul 24 03:28:13.934884 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:13.934862 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29c111eb-b99b-4c47-94ec-373c4f1e656a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-s2vht\" (UID: \"29c111eb-b99b-4c47-94ec-373c4f1e656a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht" Jul 24 03:28:13.979576 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:13.979533 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:14.035547 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:14.035512 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29c111eb-b99b-4c47-94ec-373c4f1e656a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-s2vht\" (UID: \"29c111eb-b99b-4c47-94ec-373c4f1e656a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht" Jul 24 03:28:14.035720 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:14.035555 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trcls\" (UniqueName: \"kubernetes.io/projected/29c111eb-b99b-4c47-94ec-373c4f1e656a-kube-api-access-trcls\") pod \"kuadrant-operator-controller-manager-8684c9ff49-s2vht\" (UID: \"29c111eb-b99b-4c47-94ec-373c4f1e656a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht" Jul 24 03:28:14.035894 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:14.035872 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29c111eb-b99b-4c47-94ec-373c4f1e656a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-s2vht\" (UID: \"29c111eb-b99b-4c47-94ec-373c4f1e656a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht" Jul 24 03:28:14.044383 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:14.044358 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trcls\" (UniqueName: \"kubernetes.io/projected/29c111eb-b99b-4c47-94ec-373c4f1e656a-kube-api-access-trcls\") pod \"kuadrant-operator-controller-manager-8684c9ff49-s2vht\" (UID: \"29c111eb-b99b-4c47-94ec-373c4f1e656a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht" Jul 24 03:28:14.210413 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:14.210331 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht" Jul 24 03:28:14.264541 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:14.264401 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:14.353969 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:14.353807 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht"] Jul 24 03:28:14.356872 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:28:14.356837 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c111eb_b99b_4c47_94ec_373c4f1e656a.slice/crio-b9e960e06ff7ac22789da4988f8e6b591d81ce5c361c94103b2d8225c169dce6 WatchSource:0}: Error finding container b9e960e06ff7ac22789da4988f8e6b591d81ce5c361c94103b2d8225c169dce6: Status 404 returned error can't find the container with id b9e960e06ff7ac22789da4988f8e6b591d81ce5c361c94103b2d8225c169dce6 Jul 24 03:28:14.393320 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:14.393283 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:14.527390 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:14.527309 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht" event={"ID":"29c111eb-b99b-4c47-94ec-373c4f1e656a","Type":"ContainerStarted","Data":"b9e960e06ff7ac22789da4988f8e6b591d81ce5c361c94103b2d8225c169dce6"} Jul 24 03:28:14.543569 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:14.543532 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:17.044864 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:17.044828 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:17.317555 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:17.317483 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-kksmh"] Jul 24 03:28:17.321234 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:17.321211 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kksmh" Jul 24 03:28:17.325309 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:17.325284 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-vx9mb\"" Jul 24 03:28:17.334948 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:17.334920 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-kksmh"] Jul 24 03:28:17.364775 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:17.364744 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv7w6\" (UniqueName: \"kubernetes.io/projected/5848e713-d744-4ecd-baf7-40d3bdf70db6-kube-api-access-vv7w6\") pod \"authorino-operator-75cc8d5cb7-kksmh\" (UID: \"5848e713-d744-4ecd-baf7-40d3bdf70db6\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-kksmh" Jul 24 03:28:17.465822 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:17.465788 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv7w6\" (UniqueName: \"kubernetes.io/projected/5848e713-d744-4ecd-baf7-40d3bdf70db6-kube-api-access-vv7w6\") pod \"authorino-operator-75cc8d5cb7-kksmh\" (UID: \"5848e713-d744-4ecd-baf7-40d3bdf70db6\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-kksmh" Jul 24 03:28:17.480634 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:17.480599 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv7w6\" (UniqueName: \"kubernetes.io/projected/5848e713-d744-4ecd-baf7-40d3bdf70db6-kube-api-access-vv7w6\") pod \"authorino-operator-75cc8d5cb7-kksmh\" (UID: \"5848e713-d744-4ecd-baf7-40d3bdf70db6\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-kksmh" Jul 24 03:28:17.640168 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:17.640133 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kksmh" Jul 24 03:28:17.781932 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:17.781905 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-kksmh"] Jul 24 03:28:17.784389 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:28:17.784357 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5848e713_d744_4ecd_baf7_40d3bdf70db6.slice/crio-1269d4fc6e9701a26c3e18490c5523a2b6389a1ad996fcb6f025c8bcd471fe97 WatchSource:0}: Error finding container 1269d4fc6e9701a26c3e18490c5523a2b6389a1ad996fcb6f025c8bcd471fe97: Status 404 returned error can't find the container with id 1269d4fc6e9701a26c3e18490c5523a2b6389a1ad996fcb6f025c8bcd471fe97 Jul 24 03:28:18.500358 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:18.500329 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-twsqn" Jul 24 03:28:18.549277 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:18.549203 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kksmh" event={"ID":"5848e713-d744-4ecd-baf7-40d3bdf70db6","Type":"ContainerStarted","Data":"1269d4fc6e9701a26c3e18490c5523a2b6389a1ad996fcb6f025c8bcd471fe97"} Jul 24 03:28:18.592133 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:18.592097 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:18.688482 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:18.688429 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:19.557735 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:19.557698 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht" event={"ID":"29c111eb-b99b-4c47-94ec-373c4f1e656a","Type":"ContainerStarted","Data":"a90ed9779906ef7f18bc7d41e75c33f29de0354f5c8accfef332be3baa537c6f"} Jul 24 03:28:19.558210 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:19.557833 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht" Jul 24 03:28:19.582161 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:19.582107 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht" podStartSLOduration=1.877576009 podStartE2EDuration="6.582084136s" totalImagesPullingTime="4.704508127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:28:19.579683189 +0000 UTC m=+649.019102648" watchObservedRunningTime="2026-07-24 03:28:19.582084136 +0000 UTC m=+649.021503665" Jul 24 03:28:21.569716 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:21.569684 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kksmh" event={"ID":"5848e713-d744-4ecd-baf7-40d3bdf70db6","Type":"ContainerStarted","Data":"d68e1943ed06a40a5a4ced8ccf5fda991501ccac5b2e189c8865b9c9392a3517"} Jul 24 03:28:21.570114 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:21.569822 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kksmh" Jul 24 03:28:21.589656 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:21.589602 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kksmh" podStartSLOduration=1.539897673 podStartE2EDuration="4.589586872s" totalImagesPullingTime="3.049689199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:28:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:28:21.587838758 +0000 UTC m=+651.027258228" watchObservedRunningTime="2026-07-24 03:28:21.589586872 +0000 UTC m=+651.029006394" Jul 24 03:28:21.980786 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:21.980750 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:22.007229 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:22.007199 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-xq4p7"] Jul 24 03:28:22.010830 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:22.010808 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xq4p7" Jul 24 03:28:22.013114 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:22.013090 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 24 03:28:22.013243 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:22.013189 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 24 03:28:22.013243 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:22.013225 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4hl77\"" Jul 24 03:28:22.019202 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:22.019182 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-xq4p7"] Jul 24 03:28:22.082593 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:22.082555 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:22.110624 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:22.110596 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/dbc8994c-8f77-41e7-ada1-8aaa9a85de74-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-xq4p7\" (UID: \"dbc8994c-8f77-41e7-ada1-8aaa9a85de74\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xq4p7" Jul 24 03:28:22.110781 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:22.110629 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dbc8994c-8f77-41e7-ada1-8aaa9a85de74-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-xq4p7\" (UID: \"dbc8994c-8f77-41e7-ada1-8aaa9a85de74\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xq4p7" Jul 24 03:28:22.110781 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:22.110672 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5nrh\" (UniqueName: \"kubernetes.io/projected/dbc8994c-8f77-41e7-ada1-8aaa9a85de74-kube-api-access-l5nrh\") pod \"kuadrant-console-plugin-66b5774985-xq4p7\" (UID: \"dbc8994c-8f77-41e7-ada1-8aaa9a85de74\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xq4p7" Jul 24 03:28:22.211512 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:22.211481 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/dbc8994c-8f77-41e7-ada1-8aaa9a85de74-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-xq4p7\" (UID: \"dbc8994c-8f77-41e7-ada1-8aaa9a85de74\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xq4p7" Jul 24 03:28:22.211512 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:22.211515 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dbc8994c-8f77-41e7-ada1-8aaa9a85de74-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-xq4p7\" (UID: \"dbc8994c-8f77-41e7-ada1-8aaa9a85de74\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xq4p7" Jul 24 03:28:22.211729 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:22.211558 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5nrh\" (UniqueName: \"kubernetes.io/projected/dbc8994c-8f77-41e7-ada1-8aaa9a85de74-kube-api-access-l5nrh\") pod \"kuadrant-console-plugin-66b5774985-xq4p7\" (UID: \"dbc8994c-8f77-41e7-ada1-8aaa9a85de74\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xq4p7" Jul 24 03:28:22.212174 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:22.212154 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dbc8994c-8f77-41e7-ada1-8aaa9a85de74-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-xq4p7\" (UID: \"dbc8994c-8f77-41e7-ada1-8aaa9a85de74\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xq4p7" Jul 24 03:28:22.214024 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:22.213992 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/dbc8994c-8f77-41e7-ada1-8aaa9a85de74-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-xq4p7\" (UID: \"dbc8994c-8f77-41e7-ada1-8aaa9a85de74\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xq4p7" Jul 24 03:28:22.218216 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:22.218188 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:22.220389 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:22.220360 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5nrh\" (UniqueName: \"kubernetes.io/projected/dbc8994c-8f77-41e7-ada1-8aaa9a85de74-kube-api-access-l5nrh\") pod \"kuadrant-console-plugin-66b5774985-xq4p7\" (UID: \"dbc8994c-8f77-41e7-ada1-8aaa9a85de74\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xq4p7" Jul 24 03:28:22.320401 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:22.320320 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xq4p7" Jul 24 03:28:22.461371 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:22.461336 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-xq4p7"] Jul 24 03:28:22.463543 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:28:22.463517 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbc8994c_8f77_41e7_ada1_8aaa9a85de74.slice/crio-cb36fccf3a60f02a9fdfdc8375bf250f29cca11887e45797f82692fc86e5d6b9 WatchSource:0}: Error finding container cb36fccf3a60f02a9fdfdc8375bf250f29cca11887e45797f82692fc86e5d6b9: Status 404 returned error can't find the container with id cb36fccf3a60f02a9fdfdc8375bf250f29cca11887e45797f82692fc86e5d6b9 Jul 24 03:28:22.464972 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:22.464955 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 03:28:22.520566 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:22.520535 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8g9dl" Jul 24 03:28:22.576285 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:22.576197 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xq4p7" event={"ID":"dbc8994c-8f77-41e7-ada1-8aaa9a85de74","Type":"ContainerStarted","Data":"cb36fccf3a60f02a9fdfdc8375bf250f29cca11887e45797f82692fc86e5d6b9"} Jul 24 03:28:25.383532 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:25.383492 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:25.446227 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:25.446120 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:30.566331 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:30.566298 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht" Jul 24 03:28:32.341195 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.341161 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht"] Jul 24 03:28:32.341628 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.341371 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht" podUID="29c111eb-b99b-4c47-94ec-373c4f1e656a" containerName="manager" containerID="cri-o://a90ed9779906ef7f18bc7d41e75c33f29de0354f5c8accfef332be3baa537c6f" gracePeriod=2 Jul 24 03:28:32.344680 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.344627 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht"] Jul 24 03:28:32.345329 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:28:32.345290 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-s2vht\" is forbidden: User \"system:node:ip-10-0-134-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-147.ec2.internal' and this object" podUID="29c111eb-b99b-4c47-94ec-373c4f1e656a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht" Jul 24 03:28:32.364362 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.364334 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5z4r"] Jul 24 03:28:32.364932 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.364907 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29c111eb-b99b-4c47-94ec-373c4f1e656a" containerName="manager" Jul 24 03:28:32.365065 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.364937 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="29c111eb-b99b-4c47-94ec-373c4f1e656a" containerName="manager" Jul 24 03:28:32.365065 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.365060 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="29c111eb-b99b-4c47-94ec-373c4f1e656a" containerName="manager" Jul 24 03:28:32.387965 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.387932 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5z4r"] Jul 24 03:28:32.388082 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.388032 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8g9dl"] Jul 24 03:28:32.388220 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.388202 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5z4r" Jul 24 03:28:32.388346 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.388309 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8g9dl" podUID="06c5c9db-7be6-49e3-850d-ddc5658c992c" containerName="manager" containerID="cri-o://d8e7671fc6464a22b2ab48d6c85a0fcaea901f1b2e7990b68429463720fab6a9" gracePeriod=2 Jul 24 03:28:32.394541 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.394491 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8g9dl"] Jul 24 03:28:32.424985 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.424958 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dmrst"] Jul 24 03:28:32.425422 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.425401 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06c5c9db-7be6-49e3-850d-ddc5658c992c" containerName="manager" Jul 24 03:28:32.425422 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.425423 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="06c5c9db-7be6-49e3-850d-ddc5658c992c" containerName="manager" Jul 24 03:28:32.425635 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.425574 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="06c5c9db-7be6-49e3-850d-ddc5658c992c" containerName="manager" Jul 24 03:28:32.431398 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.430521 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dmrst" Jul 24 03:28:32.431398 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:28:32.431307 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-s2vht\" is forbidden: User \"system:node:ip-10-0-134-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-147.ec2.internal' and this object" podUID="29c111eb-b99b-4c47-94ec-373c4f1e656a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht" Jul 24 03:28:32.433427 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:28:32.433394 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8g9dl\" is forbidden: User \"system:node:ip-10-0-134-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-147.ec2.internal' and this object" podUID="06c5c9db-7be6-49e3-850d-ddc5658c992c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8g9dl" Jul 24 03:28:32.445494 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.444503 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dmrst"] Jul 24 03:28:32.459251 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:28:32.459221 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-s2vht\" is forbidden: User \"system:node:ip-10-0-134-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-147.ec2.internal' and this object" podUID="29c111eb-b99b-4c47-94ec-373c4f1e656a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht" Jul 24 03:28:32.505571 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.505528 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/826bff24-d363-4391-8ee8-9cd110248852-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-w5z4r\" (UID: \"826bff24-d363-4391-8ee8-9cd110248852\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5z4r" Jul 24 03:28:32.505708 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.505648 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twqjg\" (UniqueName: \"kubernetes.io/projected/826bff24-d363-4391-8ee8-9cd110248852-kube-api-access-twqjg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-w5z4r\" (UID: \"826bff24-d363-4391-8ee8-9cd110248852\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5z4r" Jul 24 03:28:32.579087 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.579057 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kksmh" Jul 24 03:28:32.583409 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:28:32.583375 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8g9dl\" is forbidden: User \"system:node:ip-10-0-134-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-147.ec2.internal' and this object" podUID="06c5c9db-7be6-49e3-850d-ddc5658c992c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8g9dl" Jul 24 03:28:32.606928 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.606864 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd88m\" (UniqueName: \"kubernetes.io/projected/fc166356-d039-4390-9c9e-8b117b744b3f-kube-api-access-hd88m\") pod \"limitador-operator-controller-manager-859d7b78bd-dmrst\" (UID: \"fc166356-d039-4390-9c9e-8b117b744b3f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dmrst" Jul 24 03:28:32.607069 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.606940 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/826bff24-d363-4391-8ee8-9cd110248852-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-w5z4r\" (UID: \"826bff24-d363-4391-8ee8-9cd110248852\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5z4r" Jul 24 03:28:32.607069 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.606994 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twqjg\" (UniqueName: \"kubernetes.io/projected/826bff24-d363-4391-8ee8-9cd110248852-kube-api-access-twqjg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-w5z4r\" (UID: \"826bff24-d363-4391-8ee8-9cd110248852\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5z4r" Jul 24 03:28:32.607364 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.607340 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/826bff24-d363-4391-8ee8-9cd110248852-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-w5z4r\" (UID: \"826bff24-d363-4391-8ee8-9cd110248852\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5z4r" Jul 24 03:28:32.625391 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.625359 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twqjg\" (UniqueName: \"kubernetes.io/projected/826bff24-d363-4391-8ee8-9cd110248852-kube-api-access-twqjg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-w5z4r\" (UID: \"826bff24-d363-4391-8ee8-9cd110248852\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5z4r" Jul 24 03:28:32.625551 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:28:32.625470 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-s2vht\" is forbidden: User \"system:node:ip-10-0-134-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-147.ec2.internal' and this object" podUID="29c111eb-b99b-4c47-94ec-373c4f1e656a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht" Jul 24 03:28:32.707449 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.707417 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd88m\" (UniqueName: \"kubernetes.io/projected/fc166356-d039-4390-9c9e-8b117b744b3f-kube-api-access-hd88m\") pod \"limitador-operator-controller-manager-859d7b78bd-dmrst\" (UID: \"fc166356-d039-4390-9c9e-8b117b744b3f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dmrst" Jul 24 03:28:32.721168 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.721146 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd88m\" (UniqueName: \"kubernetes.io/projected/fc166356-d039-4390-9c9e-8b117b744b3f-kube-api-access-hd88m\") pod \"limitador-operator-controller-manager-859d7b78bd-dmrst\" (UID: \"fc166356-d039-4390-9c9e-8b117b744b3f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dmrst" Jul 24 03:28:32.778141 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.778101 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5z4r" Jul 24 03:28:32.787226 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:32.787199 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dmrst" Jul 24 03:28:33.519380 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:33.519321 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8g9dl" podUID="06c5c9db-7be6-49e3-850d-ddc5658c992c" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.30:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 03:28:33.835720 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:33.835613 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:33.940975 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:33.940938 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:36.420915 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:36.420875 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:36.533165 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:36.533133 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:39.583738 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:39.583702 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:39.685040 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:39.685001 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:43.188632 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.188593 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:43.289412 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.289337 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:43.455545 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.455517 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dmrst"] Jul 24 03:28:43.458273 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:28:43.458248 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc166356_d039_4390_9c9e_8b117b744b3f.slice/crio-3fecae603b851fd7764aedff0e30cb367e5dd1372f57481b003ab61a938e6c66 WatchSource:0}: Error finding container 3fecae603b851fd7764aedff0e30cb367e5dd1372f57481b003ab61a938e6c66: Status 404 returned error can't find the container with id 3fecae603b851fd7764aedff0e30cb367e5dd1372f57481b003ab61a938e6c66 Jul 24 03:28:43.464144 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.464128 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht" Jul 24 03:28:43.466145 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:28:43.466110 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-s2vht\" is forbidden: User \"system:node:ip-10-0-134-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-147.ec2.internal' and this object" podUID="29c111eb-b99b-4c47-94ec-373c4f1e656a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht" Jul 24 03:28:43.466875 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.466854 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8g9dl" Jul 24 03:28:43.468845 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:28:43.468821 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-s2vht\" is forbidden: User \"system:node:ip-10-0-134-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-147.ec2.internal' and this object" podUID="29c111eb-b99b-4c47-94ec-373c4f1e656a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht" Jul 24 03:28:43.470438 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:28:43.470395 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8g9dl\" is forbidden: User \"system:node:ip-10-0-134-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-147.ec2.internal' and this object" podUID="06c5c9db-7be6-49e3-850d-ddc5658c992c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8g9dl" Jul 24 03:28:43.481527 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.481507 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5z4r"] Jul 24 03:28:43.493626 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:28:43.493599 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826bff24_d363_4391_8ee8_9cd110248852.slice/crio-fedfcd77a6b4dc7be70e8cec84f77fd99c2943357dc66e4af26b06036e6b3091 WatchSource:0}: Error finding container fedfcd77a6b4dc7be70e8cec84f77fd99c2943357dc66e4af26b06036e6b3091: Status 404 returned error can't find the container with id fedfcd77a6b4dc7be70e8cec84f77fd99c2943357dc66e4af26b06036e6b3091 Jul 24 03:28:43.513993 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.513972 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trcls\" (UniqueName: \"kubernetes.io/projected/29c111eb-b99b-4c47-94ec-373c4f1e656a-kube-api-access-trcls\") pod \"29c111eb-b99b-4c47-94ec-373c4f1e656a\" (UID: \"29c111eb-b99b-4c47-94ec-373c4f1e656a\") " Jul 24 03:28:43.514093 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.514077 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29c111eb-b99b-4c47-94ec-373c4f1e656a-extensions-socket-volume\") pod \"29c111eb-b99b-4c47-94ec-373c4f1e656a\" (UID: \"29c111eb-b99b-4c47-94ec-373c4f1e656a\") " Jul 24 03:28:43.514140 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.514126 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzkvb\" (UniqueName: \"kubernetes.io/projected/06c5c9db-7be6-49e3-850d-ddc5658c992c-kube-api-access-nzkvb\") pod \"06c5c9db-7be6-49e3-850d-ddc5658c992c\" (UID: \"06c5c9db-7be6-49e3-850d-ddc5658c992c\") " Jul 24 03:28:43.514705 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.514673 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c111eb-b99b-4c47-94ec-373c4f1e656a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "29c111eb-b99b-4c47-94ec-373c4f1e656a" (UID: "29c111eb-b99b-4c47-94ec-373c4f1e656a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:28:43.516206 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.516165 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c111eb-b99b-4c47-94ec-373c4f1e656a-kube-api-access-trcls" (OuterVolumeSpecName: "kube-api-access-trcls") pod "29c111eb-b99b-4c47-94ec-373c4f1e656a" (UID: "29c111eb-b99b-4c47-94ec-373c4f1e656a"). InnerVolumeSpecName "kube-api-access-trcls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:28:43.516549 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.516521 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c5c9db-7be6-49e3-850d-ddc5658c992c-kube-api-access-nzkvb" (OuterVolumeSpecName: "kube-api-access-nzkvb") pod "06c5c9db-7be6-49e3-850d-ddc5658c992c" (UID: "06c5c9db-7be6-49e3-850d-ddc5658c992c"). InnerVolumeSpecName "kube-api-access-nzkvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:28:43.615172 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.615135 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trcls\" (UniqueName: \"kubernetes.io/projected/29c111eb-b99b-4c47-94ec-373c4f1e656a-kube-api-access-trcls\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:28:43.615172 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.615169 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29c111eb-b99b-4c47-94ec-373c4f1e656a-extensions-socket-volume\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:28:43.615172 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.615181 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzkvb\" (UniqueName: \"kubernetes.io/projected/06c5c9db-7be6-49e3-850d-ddc5658c992c-kube-api-access-nzkvb\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:28:43.695158 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.695067 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dmrst" event={"ID":"fc166356-d039-4390-9c9e-8b117b744b3f","Type":"ContainerStarted","Data":"0cb6d98308319a6056d7a66ba888ee18efa8eabd5b735a46ac62a20c0dfc709c"} Jul 24 03:28:43.695158 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.695130 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dmrst" event={"ID":"fc166356-d039-4390-9c9e-8b117b744b3f","Type":"ContainerStarted","Data":"3fecae603b851fd7764aedff0e30cb367e5dd1372f57481b003ab61a938e6c66"} Jul 24 03:28:43.695367 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.695177 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dmrst" Jul 24 03:28:43.697004 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.696975 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5z4r" event={"ID":"826bff24-d363-4391-8ee8-9cd110248852","Type":"ContainerStarted","Data":"09a817f40476a30cee0f5d53f761e3f83c3ae53be6da9f3a7d37708f6db3c1c9"} Jul 24 03:28:43.697152 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.697034 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5z4r" Jul 24 03:28:43.697152 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.697058 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5z4r" event={"ID":"826bff24-d363-4391-8ee8-9cd110248852","Type":"ContainerStarted","Data":"fedfcd77a6b4dc7be70e8cec84f77fd99c2943357dc66e4af26b06036e6b3091"} Jul 24 03:28:43.697330 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:28:43.697302 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-s2vht\" is forbidden: User \"system:node:ip-10-0-134-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-147.ec2.internal' and this object" podUID="29c111eb-b99b-4c47-94ec-373c4f1e656a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht" Jul 24 03:28:43.698528 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.698500 2579 generic.go:356] "Generic (PLEG): container finished" podID="06c5c9db-7be6-49e3-850d-ddc5658c992c" containerID="d8e7671fc6464a22b2ab48d6c85a0fcaea901f1b2e7990b68429463720fab6a9" exitCode=0 Jul 24 03:28:43.698643 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.698550 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8g9dl" Jul 24 03:28:43.698643 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.698589 2579 scope.go:117] "RemoveContainer" containerID="d8e7671fc6464a22b2ab48d6c85a0fcaea901f1b2e7990b68429463720fab6a9" Jul 24 03:28:43.699142 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:28:43.699118 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8g9dl\" is forbidden: User \"system:node:ip-10-0-134-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-147.ec2.internal' and this object" podUID="06c5c9db-7be6-49e3-850d-ddc5658c992c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8g9dl" Jul 24 03:28:43.700167 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.700145 2579 generic.go:356] "Generic (PLEG): container finished" podID="29c111eb-b99b-4c47-94ec-373c4f1e656a" containerID="a90ed9779906ef7f18bc7d41e75c33f29de0354f5c8accfef332be3baa537c6f" exitCode=0 Jul 24 03:28:43.700269 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.700205 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht" Jul 24 03:28:43.702804 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.702784 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xq4p7" event={"ID":"dbc8994c-8f77-41e7-ada1-8aaa9a85de74","Type":"ContainerStarted","Data":"4807a9c54da29a73af422fc6b9b46ffe3bd654c39cf3140d745a6959bdddebdb"} Jul 24 03:28:43.707678 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.707660 2579 scope.go:117] "RemoveContainer" containerID="d8e7671fc6464a22b2ab48d6c85a0fcaea901f1b2e7990b68429463720fab6a9" Jul 24 03:28:43.707946 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:28:43.707922 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8e7671fc6464a22b2ab48d6c85a0fcaea901f1b2e7990b68429463720fab6a9\": container with ID starting with d8e7671fc6464a22b2ab48d6c85a0fcaea901f1b2e7990b68429463720fab6a9 not found: ID does not exist" containerID="d8e7671fc6464a22b2ab48d6c85a0fcaea901f1b2e7990b68429463720fab6a9" Jul 24 03:28:43.707997 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.707979 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8e7671fc6464a22b2ab48d6c85a0fcaea901f1b2e7990b68429463720fab6a9"} err="failed to get container status \"d8e7671fc6464a22b2ab48d6c85a0fcaea901f1b2e7990b68429463720fab6a9\": rpc error: code = NotFound desc = could not find container \"d8e7671fc6464a22b2ab48d6c85a0fcaea901f1b2e7990b68429463720fab6a9\": container with ID starting with d8e7671fc6464a22b2ab48d6c85a0fcaea901f1b2e7990b68429463720fab6a9 not found: ID does not exist" Jul 24 03:28:43.708042 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.707996 2579 scope.go:117] "RemoveContainer" containerID="a90ed9779906ef7f18bc7d41e75c33f29de0354f5c8accfef332be3baa537c6f" Jul 24 03:28:43.715995 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.715978 2579 scope.go:117] "RemoveContainer" containerID="a90ed9779906ef7f18bc7d41e75c33f29de0354f5c8accfef332be3baa537c6f" Jul 24 03:28:43.716294 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:28:43.716269 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a90ed9779906ef7f18bc7d41e75c33f29de0354f5c8accfef332be3baa537c6f\": container with ID starting with a90ed9779906ef7f18bc7d41e75c33f29de0354f5c8accfef332be3baa537c6f not found: ID does not exist" containerID="a90ed9779906ef7f18bc7d41e75c33f29de0354f5c8accfef332be3baa537c6f" Jul 24 03:28:43.716374 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.716299 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a90ed9779906ef7f18bc7d41e75c33f29de0354f5c8accfef332be3baa537c6f"} err="failed to get container status \"a90ed9779906ef7f18bc7d41e75c33f29de0354f5c8accfef332be3baa537c6f\": rpc error: code = NotFound desc = could not find container \"a90ed9779906ef7f18bc7d41e75c33f29de0354f5c8accfef332be3baa537c6f\": container with ID starting with a90ed9779906ef7f18bc7d41e75c33f29de0354f5c8accfef332be3baa537c6f not found: ID does not exist" Jul 24 03:28:43.720890 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.720853 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dmrst" podStartSLOduration=11.720841441 podStartE2EDuration="11.720841441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:28:43.719740252 +0000 UTC m=+673.159159711" watchObservedRunningTime="2026-07-24 03:28:43.720841441 +0000 UTC m=+673.160260901" Jul 24 03:28:43.735589 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.735542 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xq4p7" podStartSLOduration=1.831200712 podStartE2EDuration="22.735531254s" totalImagesPullingTime="20.904330542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:28:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:28:43.735286488 +0000 UTC m=+673.174705951" watchObservedRunningTime="2026-07-24 03:28:43.735531254 +0000 UTC m=+673.174950715" Jul 24 03:28:43.737410 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:28:43.737383 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8g9dl\" is forbidden: User \"system:node:ip-10-0-134-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-147.ec2.internal' and this object" podUID="06c5c9db-7be6-49e3-850d-ddc5658c992c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8g9dl" Jul 24 03:28:43.758412 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:43.758372 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5z4r" podStartSLOduration=11.758358422 podStartE2EDuration="11.758358422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:28:43.758179392 +0000 UTC m=+673.197598872" watchObservedRunningTime="2026-07-24 03:28:43.758358422 +0000 UTC m=+673.197777882" Jul 24 03:28:43.760277 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:28:43.760249 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-s2vht\" is forbidden: User \"system:node:ip-10-0-134-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-147.ec2.internal' and this object" podUID="29c111eb-b99b-4c47-94ec-373c4f1e656a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-s2vht" Jul 24 03:28:45.033149 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:45.033107 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06c5c9db-7be6-49e3-850d-ddc5658c992c" path="/var/lib/kubelet/pods/06c5c9db-7be6-49e3-850d-ddc5658c992c/volumes" Jul 24 03:28:45.033664 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:45.033602 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29c111eb-b99b-4c47-94ec-373c4f1e656a" path="/var/lib/kubelet/pods/29c111eb-b99b-4c47-94ec-373c4f1e656a/volumes" Jul 24 03:28:45.331531 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:45.331420 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:45.396281 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:45.396248 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:46.146488 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:46.146435 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:46.249684 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:46.249654 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:46.347681 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:46.347610 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:46.896228 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:46.896199 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:46.952858 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:46.952828 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:48.593499 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:48.593451 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5z4r"] Jul 24 03:28:48.593922 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:48.593660 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5z4r" podUID="826bff24-d363-4391-8ee8-9cd110248852" containerName="manager" containerID="cri-o://09a817f40476a30cee0f5d53f761e3f83c3ae53be6da9f3a7d37708f6db3c1c9" gracePeriod=10 Jul 24 03:28:48.595800 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:48.595774 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5z4r" Jul 24 03:28:48.732982 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:48.732954 2579 generic.go:356] "Generic (PLEG): container finished" podID="826bff24-d363-4391-8ee8-9cd110248852" containerID="09a817f40476a30cee0f5d53f761e3f83c3ae53be6da9f3a7d37708f6db3c1c9" exitCode=0 Jul 24 03:28:48.733115 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:48.733028 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5z4r" event={"ID":"826bff24-d363-4391-8ee8-9cd110248852","Type":"ContainerDied","Data":"09a817f40476a30cee0f5d53f761e3f83c3ae53be6da9f3a7d37708f6db3c1c9"} Jul 24 03:28:48.845204 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:48.845145 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5z4r" Jul 24 03:28:48.856045 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:48.856022 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twqjg\" (UniqueName: \"kubernetes.io/projected/826bff24-d363-4391-8ee8-9cd110248852-kube-api-access-twqjg\") pod \"826bff24-d363-4391-8ee8-9cd110248852\" (UID: \"826bff24-d363-4391-8ee8-9cd110248852\") " Jul 24 03:28:48.856163 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:48.856079 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/826bff24-d363-4391-8ee8-9cd110248852-extensions-socket-volume\") pod \"826bff24-d363-4391-8ee8-9cd110248852\" (UID: \"826bff24-d363-4391-8ee8-9cd110248852\") " Jul 24 03:28:48.856484 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:48.856436 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/826bff24-d363-4391-8ee8-9cd110248852-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "826bff24-d363-4391-8ee8-9cd110248852" (UID: "826bff24-d363-4391-8ee8-9cd110248852"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:28:48.858216 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:48.858195 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/826bff24-d363-4391-8ee8-9cd110248852-kube-api-access-twqjg" (OuterVolumeSpecName: "kube-api-access-twqjg") pod "826bff24-d363-4391-8ee8-9cd110248852" (UID: "826bff24-d363-4391-8ee8-9cd110248852"). InnerVolumeSpecName "kube-api-access-twqjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:28:48.956984 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:48.956952 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/826bff24-d363-4391-8ee8-9cd110248852-extensions-socket-volume\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:28:48.956984 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:48.956979 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twqjg\" (UniqueName: \"kubernetes.io/projected/826bff24-d363-4391-8ee8-9cd110248852-kube-api-access-twqjg\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:28:49.739794 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:49.739770 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5z4r" Jul 24 03:28:49.739794 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:49.739784 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5z4r" event={"ID":"826bff24-d363-4391-8ee8-9cd110248852","Type":"ContainerDied","Data":"fedfcd77a6b4dc7be70e8cec84f77fd99c2943357dc66e4af26b06036e6b3091"} Jul 24 03:28:49.740304 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:49.739841 2579 scope.go:117] "RemoveContainer" containerID="09a817f40476a30cee0f5d53f761e3f83c3ae53be6da9f3a7d37708f6db3c1c9" Jul 24 03:28:49.758912 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:49.758887 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5z4r"] Jul 24 03:28:49.763834 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:49.763813 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-w5z4r"] Jul 24 03:28:51.033678 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:51.033644 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="826bff24-d363-4391-8ee8-9cd110248852" path="/var/lib/kubelet/pods/826bff24-d363-4391-8ee8-9cd110248852/volumes" Jul 24 03:28:51.673780 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:51.673749 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:51.768598 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:51.768557 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:54.716494 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:54.716435 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dmrst" Jul 24 03:28:57.563705 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:57.563672 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:57.612156 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:57.612120 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:28:57.667878 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:28:57.667845 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:29:05.052801 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.052767 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4"] Jul 24 03:29:05.053198 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.053132 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="826bff24-d363-4391-8ee8-9cd110248852" containerName="manager" Jul 24 03:29:05.053198 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.053144 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="826bff24-d363-4391-8ee8-9cd110248852" containerName="manager" Jul 24 03:29:05.053270 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.053213 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="826bff24-d363-4391-8ee8-9cd110248852" containerName="manager" Jul 24 03:29:05.073455 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.073419 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:29:05.201301 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.201264 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4"] Jul 24 03:29:05.201475 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.201387 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.203657 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.203637 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5l9bf\"" Jul 24 03:29:05.296910 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.296874 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6304eb4-8cda-4f2b-8d31-b2da9c27a10a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8s6w4\" (UID: \"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.296910 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.296910 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hhkt\" (UniqueName: \"kubernetes.io/projected/e6304eb4-8cda-4f2b-8d31-b2da9c27a10a-kube-api-access-8hhkt\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8s6w4\" (UID: \"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.297150 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.296939 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6304eb4-8cda-4f2b-8d31-b2da9c27a10a-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8s6w4\" (UID: \"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.297150 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.297044 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6304eb4-8cda-4f2b-8d31-b2da9c27a10a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8s6w4\" (UID: \"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.297150 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.297076 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6304eb4-8cda-4f2b-8d31-b2da9c27a10a-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8s6w4\" (UID: \"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.297150 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.297103 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6304eb4-8cda-4f2b-8d31-b2da9c27a10a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8s6w4\" (UID: \"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.297352 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.297160 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6304eb4-8cda-4f2b-8d31-b2da9c27a10a-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8s6w4\" (UID: \"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.297352 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.297185 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6304eb4-8cda-4f2b-8d31-b2da9c27a10a-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8s6w4\" (UID: \"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.297352 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.297209 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6304eb4-8cda-4f2b-8d31-b2da9c27a10a-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8s6w4\" (UID: \"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.397888 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.397857 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6304eb4-8cda-4f2b-8d31-b2da9c27a10a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8s6w4\" (UID: \"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.398048 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.397896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6304eb4-8cda-4f2b-8d31-b2da9c27a10a-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8s6w4\" (UID: \"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.398048 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.397928 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6304eb4-8cda-4f2b-8d31-b2da9c27a10a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8s6w4\" (UID: \"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.398048 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.397963 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6304eb4-8cda-4f2b-8d31-b2da9c27a10a-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8s6w4\" (UID: \"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.398048 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.397995 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6304eb4-8cda-4f2b-8d31-b2da9c27a10a-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8s6w4\" (UID: \"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.398048 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.398030 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6304eb4-8cda-4f2b-8d31-b2da9c27a10a-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8s6w4\" (UID: \"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.398309 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.398089 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6304eb4-8cda-4f2b-8d31-b2da9c27a10a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8s6w4\" (UID: \"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.398309 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.398115 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hhkt\" (UniqueName: \"kubernetes.io/projected/e6304eb4-8cda-4f2b-8d31-b2da9c27a10a-kube-api-access-8hhkt\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8s6w4\" (UID: \"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.398309 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.398148 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6304eb4-8cda-4f2b-8d31-b2da9c27a10a-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8s6w4\" (UID: \"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.398489 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.398379 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6304eb4-8cda-4f2b-8d31-b2da9c27a10a-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8s6w4\" (UID: \"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.398554 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.398495 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6304eb4-8cda-4f2b-8d31-b2da9c27a10a-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8s6w4\" (UID: \"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.398636 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.398617 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6304eb4-8cda-4f2b-8d31-b2da9c27a10a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8s6w4\" (UID: \"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.398861 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.398839 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6304eb4-8cda-4f2b-8d31-b2da9c27a10a-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8s6w4\" (UID: \"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.399032 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.398990 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6304eb4-8cda-4f2b-8d31-b2da9c27a10a-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8s6w4\" (UID: \"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.400795 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.400770 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6304eb4-8cda-4f2b-8d31-b2da9c27a10a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8s6w4\" (UID: \"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.400893 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.400830 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6304eb4-8cda-4f2b-8d31-b2da9c27a10a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8s6w4\" (UID: \"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.406754 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.406724 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hhkt\" (UniqueName: \"kubernetes.io/projected/e6304eb4-8cda-4f2b-8d31-b2da9c27a10a-kube-api-access-8hhkt\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8s6w4\" (UID: \"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.406880 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.406863 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6304eb4-8cda-4f2b-8d31-b2da9c27a10a-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8s6w4\" (UID: \"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.479190 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.479160 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:29:05.512277 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.512246 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:05.678324 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.678244 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4"] Jul 24 03:29:05.681548 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:29:05.681522 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6304eb4_8cda_4f2b_8d31_b2da9c27a10a.slice/crio-09835198d582f108c5359681230d1667270efb18ca1f869700f031f9d51e3fa4 WatchSource:0}: Error finding container 09835198d582f108c5359681230d1667270efb18ca1f869700f031f9d51e3fa4: Status 404 returned error can't find the container with id 09835198d582f108c5359681230d1667270efb18ca1f869700f031f9d51e3fa4 Jul 24 03:29:05.683645 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.683603 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:29:05.683727 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.683699 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:29:05.683765 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.683746 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:29:05.825063 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.825035 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" event={"ID":"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a","Type":"ContainerStarted","Data":"d7083a9f0ec86e5f5a212f236c1979d12f21e585558416bf50cfe52f6bab4683"} Jul 24 03:29:05.825204 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:05.825092 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" event={"ID":"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a","Type":"ContainerStarted","Data":"09835198d582f108c5359681230d1667270efb18ca1f869700f031f9d51e3fa4"} Jul 24 03:29:06.848841 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:06.848781 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" podStartSLOduration=1.848762318 podStartE2EDuration="1.848762318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:29:06.847938844 +0000 UTC m=+696.287358305" watchObservedRunningTime="2026-07-24 03:29:06.848762318 +0000 UTC m=+696.288181779" Jul 24 03:29:07.513200 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:07.513161 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:07.518038 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:07.518013 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:07.835022 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:07.834951 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:07.835894 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:07.835864 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:29:07.880629 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:07.880597 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5h9cj"] Jul 24 03:29:07.883480 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:07.883440 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:29:07.963473 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:07.963426 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5h9cj"] Jul 24 03:29:07.963659 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:07.963551 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-5h9cj" Jul 24 03:29:07.966790 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:07.966771 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 24 03:29:07.988317 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:07.988290 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5h9cj"] Jul 24 03:29:08.017867 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:08.017828 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:29:08.021059 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:08.021031 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmrnn\" (UniqueName: \"kubernetes.io/projected/1d6b6794-c2ba-4091-8062-5446a9a9a5ed-kube-api-access-gmrnn\") pod \"limitador-limitador-64c94fd64f-5h9cj\" (UID: \"1d6b6794-c2ba-4091-8062-5446a9a9a5ed\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5h9cj" Jul 24 03:29:08.021605 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:08.021108 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1d6b6794-c2ba-4091-8062-5446a9a9a5ed-config-file\") pod \"limitador-limitador-64c94fd64f-5h9cj\" (UID: \"1d6b6794-c2ba-4091-8062-5446a9a9a5ed\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5h9cj" Jul 24 03:29:08.118256 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:08.118228 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:29:08.122421 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:08.122394 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmrnn\" (UniqueName: \"kubernetes.io/projected/1d6b6794-c2ba-4091-8062-5446a9a9a5ed-kube-api-access-gmrnn\") pod \"limitador-limitador-64c94fd64f-5h9cj\" (UID: \"1d6b6794-c2ba-4091-8062-5446a9a9a5ed\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5h9cj" Jul 24 03:29:08.122573 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:08.122442 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1d6b6794-c2ba-4091-8062-5446a9a9a5ed-config-file\") pod \"limitador-limitador-64c94fd64f-5h9cj\" (UID: \"1d6b6794-c2ba-4091-8062-5446a9a9a5ed\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5h9cj" Jul 24 03:29:08.123246 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:08.123224 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1d6b6794-c2ba-4091-8062-5446a9a9a5ed-config-file\") pod \"limitador-limitador-64c94fd64f-5h9cj\" (UID: \"1d6b6794-c2ba-4091-8062-5446a9a9a5ed\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5h9cj" Jul 24 03:29:08.132248 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:08.132220 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmrnn\" (UniqueName: \"kubernetes.io/projected/1d6b6794-c2ba-4091-8062-5446a9a9a5ed-kube-api-access-gmrnn\") pod \"limitador-limitador-64c94fd64f-5h9cj\" (UID: \"1d6b6794-c2ba-4091-8062-5446a9a9a5ed\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-5h9cj" Jul 24 03:29:08.274117 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:08.274085 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-5h9cj" Jul 24 03:29:08.462098 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:08.462076 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5h9cj"] Jul 24 03:29:08.464335 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:29:08.464308 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d6b6794_c2ba_4091_8062_5446a9a9a5ed.slice/crio-3d503495b5a06e69e6ad301239897337b152bdafd0a934f20bc97a9752d084dd WatchSource:0}: Error finding container 3d503495b5a06e69e6ad301239897337b152bdafd0a934f20bc97a9752d084dd: Status 404 returned error can't find the container with id 3d503495b5a06e69e6ad301239897337b152bdafd0a934f20bc97a9752d084dd Jul 24 03:29:08.779362 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:08.779288 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-qhtvh"] Jul 24 03:29:08.805198 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:08.805163 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:29:08.806399 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:08.806375 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-qhtvh"] Jul 24 03:29:08.806572 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:08.806532 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qhtvh" Jul 24 03:29:08.808851 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:08.808827 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wxnqx\"" Jul 24 03:29:08.842116 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:08.842079 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-5h9cj" event={"ID":"1d6b6794-c2ba-4091-8062-5446a9a9a5ed","Type":"ContainerStarted","Data":"3d503495b5a06e69e6ad301239897337b152bdafd0a934f20bc97a9752d084dd"} Jul 24 03:29:08.883169 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:08.883139 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:29:08.931084 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:08.930770 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpmbh\" (UniqueName: \"kubernetes.io/projected/2607cf44-a0ff-4b47-ae93-4f18c365b6d3-kube-api-access-dpmbh\") pod \"authorino-58cd955675-qhtvh\" (UID: \"2607cf44-a0ff-4b47-ae93-4f18c365b6d3\") " pod="kuadrant-system/authorino-58cd955675-qhtvh" Jul 24 03:29:09.031491 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:09.031406 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpmbh\" (UniqueName: \"kubernetes.io/projected/2607cf44-a0ff-4b47-ae93-4f18c365b6d3-kube-api-access-dpmbh\") pod \"authorino-58cd955675-qhtvh\" (UID: \"2607cf44-a0ff-4b47-ae93-4f18c365b6d3\") " pod="kuadrant-system/authorino-58cd955675-qhtvh" Jul 24 03:29:09.040115 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:09.040089 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpmbh\" (UniqueName: \"kubernetes.io/projected/2607cf44-a0ff-4b47-ae93-4f18c365b6d3-kube-api-access-dpmbh\") pod \"authorino-58cd955675-qhtvh\" (UID: \"2607cf44-a0ff-4b47-ae93-4f18c365b6d3\") " pod="kuadrant-system/authorino-58cd955675-qhtvh" Jul 24 03:29:09.116732 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:09.116704 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qhtvh" Jul 24 03:29:09.294515 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:09.294491 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-qhtvh"] Jul 24 03:29:09.296936 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:29:09.296905 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2607cf44_a0ff_4b47_ae93_4f18c365b6d3.slice/crio-646390f87f7eb0386dd2988d61133f0cba884de1ed80aad03fba1d658faa3ef0 WatchSource:0}: Error finding container 646390f87f7eb0386dd2988d61133f0cba884de1ed80aad03fba1d658faa3ef0: Status 404 returned error can't find the container with id 646390f87f7eb0386dd2988d61133f0cba884de1ed80aad03fba1d658faa3ef0 Jul 24 03:29:09.855928 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:09.855885 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qhtvh" event={"ID":"2607cf44-a0ff-4b47-ae93-4f18c365b6d3","Type":"ContainerStarted","Data":"646390f87f7eb0386dd2988d61133f0cba884de1ed80aad03fba1d658faa3ef0"} Jul 24 03:29:13.543980 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:13.543947 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-qhtvh"] Jul 24 03:29:14.883265 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:14.883215 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qhtvh" event={"ID":"2607cf44-a0ff-4b47-ae93-4f18c365b6d3","Type":"ContainerStarted","Data":"92c4b4677e0d3d8e817e079cdf595d524e84cc7628f0f702c656844199073d77"} Jul 24 03:29:14.883265 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:14.883217 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-qhtvh" podUID="2607cf44-a0ff-4b47-ae93-4f18c365b6d3" containerName="authorino" containerID="cri-o://92c4b4677e0d3d8e817e079cdf595d524e84cc7628f0f702c656844199073d77" gracePeriod=30 Jul 24 03:29:14.885018 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:14.884987 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-5h9cj" event={"ID":"1d6b6794-c2ba-4091-8062-5446a9a9a5ed","Type":"ContainerStarted","Data":"fba63e5aedb264c7a97f731709482f5b5d7ce7129415580869d86664531aa796"} Jul 24 03:29:14.885134 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:14.885059 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-5h9cj" Jul 24 03:29:14.898876 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:14.898836 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-qhtvh" podStartSLOduration=1.812207619 podStartE2EDuration="6.89882478s" totalImagesPullingTime="5.086617161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:29:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:29:14.897585515 +0000 UTC m=+704.337004988" watchObservedRunningTime="2026-07-24 03:29:14.89882478 +0000 UTC m=+704.338244274" Jul 24 03:29:14.913798 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:14.913754 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-5h9cj" podStartSLOduration=1.99592319 podStartE2EDuration="7.913737277s" totalImagesPullingTime="5.917814087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:29:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:29:14.912626415 +0000 UTC m=+704.352045891" watchObservedRunningTime="2026-07-24 03:29:14.913737277 +0000 UTC m=+704.353156736" Jul 24 03:29:15.139400 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:15.139344 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qhtvh" Jul 24 03:29:15.291414 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:15.291380 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpmbh\" (UniqueName: \"kubernetes.io/projected/2607cf44-a0ff-4b47-ae93-4f18c365b6d3-kube-api-access-dpmbh\") pod \"2607cf44-a0ff-4b47-ae93-4f18c365b6d3\" (UID: \"2607cf44-a0ff-4b47-ae93-4f18c365b6d3\") " Jul 24 03:29:15.293601 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:15.293575 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2607cf44-a0ff-4b47-ae93-4f18c365b6d3-kube-api-access-dpmbh" (OuterVolumeSpecName: "kube-api-access-dpmbh") pod "2607cf44-a0ff-4b47-ae93-4f18c365b6d3" (UID: "2607cf44-a0ff-4b47-ae93-4f18c365b6d3"). InnerVolumeSpecName "kube-api-access-dpmbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:29:15.392819 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:15.392741 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpmbh\" (UniqueName: \"kubernetes.io/projected/2607cf44-a0ff-4b47-ae93-4f18c365b6d3-kube-api-access-dpmbh\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:29:15.891791 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:15.891757 2579 generic.go:356] "Generic (PLEG): container finished" podID="2607cf44-a0ff-4b47-ae93-4f18c365b6d3" containerID="92c4b4677e0d3d8e817e079cdf595d524e84cc7628f0f702c656844199073d77" exitCode=0 Jul 24 03:29:15.892214 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:15.891804 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qhtvh" Jul 24 03:29:15.892214 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:15.891850 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qhtvh" event={"ID":"2607cf44-a0ff-4b47-ae93-4f18c365b6d3","Type":"ContainerDied","Data":"92c4b4677e0d3d8e817e079cdf595d524e84cc7628f0f702c656844199073d77"} Jul 24 03:29:15.892214 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:15.891889 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qhtvh" event={"ID":"2607cf44-a0ff-4b47-ae93-4f18c365b6d3","Type":"ContainerDied","Data":"646390f87f7eb0386dd2988d61133f0cba884de1ed80aad03fba1d658faa3ef0"} Jul 24 03:29:15.892214 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:15.891906 2579 scope.go:117] "RemoveContainer" containerID="92c4b4677e0d3d8e817e079cdf595d524e84cc7628f0f702c656844199073d77" Jul 24 03:29:15.900784 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:15.900768 2579 scope.go:117] "RemoveContainer" containerID="92c4b4677e0d3d8e817e079cdf595d524e84cc7628f0f702c656844199073d77" Jul 24 03:29:15.901070 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:29:15.901046 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92c4b4677e0d3d8e817e079cdf595d524e84cc7628f0f702c656844199073d77\": container with ID starting with 92c4b4677e0d3d8e817e079cdf595d524e84cc7628f0f702c656844199073d77 not found: ID does not exist" containerID="92c4b4677e0d3d8e817e079cdf595d524e84cc7628f0f702c656844199073d77" Jul 24 03:29:15.901115 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:15.901083 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c4b4677e0d3d8e817e079cdf595d524e84cc7628f0f702c656844199073d77"} err="failed to get container status \"92c4b4677e0d3d8e817e079cdf595d524e84cc7628f0f702c656844199073d77\": rpc error: code = NotFound desc = could not find container \"92c4b4677e0d3d8e817e079cdf595d524e84cc7628f0f702c656844199073d77\": container with ID starting with 92c4b4677e0d3d8e817e079cdf595d524e84cc7628f0f702c656844199073d77 not found: ID does not exist" Jul 24 03:29:15.912491 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:15.912446 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-qhtvh"] Jul 24 03:29:15.916002 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:15.915978 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-qhtvh"] Jul 24 03:29:17.033888 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:17.033856 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2607cf44-a0ff-4b47-ae93-4f18c365b6d3" path="/var/lib/kubelet/pods/2607cf44-a0ff-4b47-ae93-4f18c365b6d3/volumes" Jul 24 03:29:22.547746 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:22.547678 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5h9cj"] Jul 24 03:29:22.548167 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:22.547846 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-5h9cj" podUID="1d6b6794-c2ba-4091-8062-5446a9a9a5ed" containerName="limitador" containerID="cri-o://fba63e5aedb264c7a97f731709482f5b5d7ce7129415580869d86664531aa796" gracePeriod=30 Jul 24 03:29:22.550220 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:22.550199 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-5h9cj" Jul 24 03:29:23.096449 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:23.096420 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-5h9cj" Jul 24 03:29:23.259735 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:23.259694 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1d6b6794-c2ba-4091-8062-5446a9a9a5ed-config-file\") pod \"1d6b6794-c2ba-4091-8062-5446a9a9a5ed\" (UID: \"1d6b6794-c2ba-4091-8062-5446a9a9a5ed\") " Jul 24 03:29:23.259735 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:23.259748 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmrnn\" (UniqueName: \"kubernetes.io/projected/1d6b6794-c2ba-4091-8062-5446a9a9a5ed-kube-api-access-gmrnn\") pod \"1d6b6794-c2ba-4091-8062-5446a9a9a5ed\" (UID: \"1d6b6794-c2ba-4091-8062-5446a9a9a5ed\") " Jul 24 03:29:23.260158 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:23.260132 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d6b6794-c2ba-4091-8062-5446a9a9a5ed-config-file" (OuterVolumeSpecName: "config-file") pod "1d6b6794-c2ba-4091-8062-5446a9a9a5ed" (UID: "1d6b6794-c2ba-4091-8062-5446a9a9a5ed"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 03:29:23.261990 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:23.261965 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d6b6794-c2ba-4091-8062-5446a9a9a5ed-kube-api-access-gmrnn" (OuterVolumeSpecName: "kube-api-access-gmrnn") pod "1d6b6794-c2ba-4091-8062-5446a9a9a5ed" (UID: "1d6b6794-c2ba-4091-8062-5446a9a9a5ed"). InnerVolumeSpecName "kube-api-access-gmrnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:29:23.361133 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:23.361099 2579 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1d6b6794-c2ba-4091-8062-5446a9a9a5ed-config-file\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:29:23.361133 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:23.361130 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmrnn\" (UniqueName: \"kubernetes.io/projected/1d6b6794-c2ba-4091-8062-5446a9a9a5ed-kube-api-access-gmrnn\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:29:23.932826 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:23.932791 2579 generic.go:356] "Generic (PLEG): container finished" podID="1d6b6794-c2ba-4091-8062-5446a9a9a5ed" containerID="fba63e5aedb264c7a97f731709482f5b5d7ce7129415580869d86664531aa796" exitCode=0 Jul 24 03:29:23.933272 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:23.932840 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-5h9cj" Jul 24 03:29:23.933272 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:23.932882 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-5h9cj" event={"ID":"1d6b6794-c2ba-4091-8062-5446a9a9a5ed","Type":"ContainerDied","Data":"fba63e5aedb264c7a97f731709482f5b5d7ce7129415580869d86664531aa796"} Jul 24 03:29:23.933272 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:23.932931 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-5h9cj" event={"ID":"1d6b6794-c2ba-4091-8062-5446a9a9a5ed","Type":"ContainerDied","Data":"3d503495b5a06e69e6ad301239897337b152bdafd0a934f20bc97a9752d084dd"} Jul 24 03:29:23.933272 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:23.932952 2579 scope.go:117] "RemoveContainer" containerID="fba63e5aedb264c7a97f731709482f5b5d7ce7129415580869d86664531aa796" Jul 24 03:29:23.942332 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:23.942316 2579 scope.go:117] "RemoveContainer" containerID="fba63e5aedb264c7a97f731709482f5b5d7ce7129415580869d86664531aa796" Jul 24 03:29:23.942629 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:29:23.942606 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fba63e5aedb264c7a97f731709482f5b5d7ce7129415580869d86664531aa796\": container with ID starting with fba63e5aedb264c7a97f731709482f5b5d7ce7129415580869d86664531aa796 not found: ID does not exist" containerID="fba63e5aedb264c7a97f731709482f5b5d7ce7129415580869d86664531aa796" Jul 24 03:29:23.942710 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:23.942642 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fba63e5aedb264c7a97f731709482f5b5d7ce7129415580869d86664531aa796"} err="failed to get container status \"fba63e5aedb264c7a97f731709482f5b5d7ce7129415580869d86664531aa796\": rpc error: code = NotFound desc = could not find container \"fba63e5aedb264c7a97f731709482f5b5d7ce7129415580869d86664531aa796\": container with ID starting with fba63e5aedb264c7a97f731709482f5b5d7ce7129415580869d86664531aa796 not found: ID does not exist" Jul 24 03:29:23.956394 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:23.956372 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5h9cj"] Jul 24 03:29:23.959883 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:23.959864 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-5h9cj"] Jul 24 03:29:24.067316 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:24.067273 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:29:24.177718 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:24.177684 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:29:24.533548 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:24.533499 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4f8cw"] Jul 24 03:29:24.533974 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:24.533957 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2607cf44-a0ff-4b47-ae93-4f18c365b6d3" containerName="authorino" Jul 24 03:29:24.533974 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:24.533975 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2607cf44-a0ff-4b47-ae93-4f18c365b6d3" containerName="authorino" Jul 24 03:29:24.534122 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:24.533997 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d6b6794-c2ba-4091-8062-5446a9a9a5ed" containerName="limitador" Jul 24 03:29:24.534122 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:24.534003 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d6b6794-c2ba-4091-8062-5446a9a9a5ed" containerName="limitador" Jul 24 03:29:24.534122 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:24.534063 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2607cf44-a0ff-4b47-ae93-4f18c365b6d3" containerName="authorino" Jul 24 03:29:24.534122 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:24.534073 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d6b6794-c2ba-4091-8062-5446a9a9a5ed" containerName="limitador" Jul 24 03:29:24.536119 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:24.536093 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4f8cw" Jul 24 03:29:24.538689 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:24.538656 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 24 03:29:24.538689 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:24.538686 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 24 03:29:24.538856 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:24.538798 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-q4r76\"" Jul 24 03:29:24.539535 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:24.539523 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 03:29:24.546603 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:24.546583 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4f8cw"] Jul 24 03:29:24.573447 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:24.573423 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:29:24.672500 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:24.672436 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/86e1bda7-4a0a-428f-8cf8-76ba06dc7fb9-data\") pod \"postgres-69fb4594cf-4f8cw\" (UID: \"86e1bda7-4a0a-428f-8cf8-76ba06dc7fb9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4f8cw" Jul 24 03:29:24.672664 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:24.672506 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsgkf\" (UniqueName: \"kubernetes.io/projected/86e1bda7-4a0a-428f-8cf8-76ba06dc7fb9-kube-api-access-fsgkf\") pod \"postgres-69fb4594cf-4f8cw\" (UID: \"86e1bda7-4a0a-428f-8cf8-76ba06dc7fb9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4f8cw" Jul 24 03:29:24.773553 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:24.773520 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/86e1bda7-4a0a-428f-8cf8-76ba06dc7fb9-data\") pod \"postgres-69fb4594cf-4f8cw\" (UID: \"86e1bda7-4a0a-428f-8cf8-76ba06dc7fb9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4f8cw" Jul 24 03:29:24.773715 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:24.773558 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsgkf\" (UniqueName: \"kubernetes.io/projected/86e1bda7-4a0a-428f-8cf8-76ba06dc7fb9-kube-api-access-fsgkf\") pod \"postgres-69fb4594cf-4f8cw\" (UID: \"86e1bda7-4a0a-428f-8cf8-76ba06dc7fb9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4f8cw" Jul 24 03:29:24.773930 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:24.773909 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/86e1bda7-4a0a-428f-8cf8-76ba06dc7fb9-data\") pod \"postgres-69fb4594cf-4f8cw\" (UID: \"86e1bda7-4a0a-428f-8cf8-76ba06dc7fb9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4f8cw" Jul 24 03:29:24.782592 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:24.782572 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsgkf\" (UniqueName: \"kubernetes.io/projected/86e1bda7-4a0a-428f-8cf8-76ba06dc7fb9-kube-api-access-fsgkf\") pod \"postgres-69fb4594cf-4f8cw\" (UID: \"86e1bda7-4a0a-428f-8cf8-76ba06dc7fb9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4f8cw" Jul 24 03:29:24.846762 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:24.846687 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4f8cw" Jul 24 03:29:24.983593 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:24.983569 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4f8cw"] Jul 24 03:29:24.985479 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:29:24.985435 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e1bda7_4a0a_428f_8cf8_76ba06dc7fb9.slice/crio-dc6ad4e5d303f05d5318a7fc9aa79b755f2b05cedf1d01974c0adb9cfeec449b WatchSource:0}: Error finding container dc6ad4e5d303f05d5318a7fc9aa79b755f2b05cedf1d01974c0adb9cfeec449b: Status 404 returned error can't find the container with id dc6ad4e5d303f05d5318a7fc9aa79b755f2b05cedf1d01974c0adb9cfeec449b Jul 24 03:29:25.033971 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:25.033931 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d6b6794-c2ba-4091-8062-5446a9a9a5ed" path="/var/lib/kubelet/pods/1d6b6794-c2ba-4091-8062-5446a9a9a5ed/volumes" Jul 24 03:29:25.949528 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:25.949307 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4f8cw" event={"ID":"86e1bda7-4a0a-428f-8cf8-76ba06dc7fb9","Type":"ContainerStarted","Data":"dc6ad4e5d303f05d5318a7fc9aa79b755f2b05cedf1d01974c0adb9cfeec449b"} Jul 24 03:29:29.972966 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:29.972910 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4f8cw" event={"ID":"86e1bda7-4a0a-428f-8cf8-76ba06dc7fb9","Type":"ContainerStarted","Data":"65f3b47908cd4f476a119e623ce39846c2765ae36718998f11a6bfe22820555d"} Jul 24 03:29:29.973380 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:29.973026 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4f8cw" Jul 24 03:29:29.990404 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:29.990345 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4f8cw" podStartSLOduration=1.1312243180000001 podStartE2EDuration="5.990330437s" totalImagesPullingTime="4.859106119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:29:29.989437837 +0000 UTC m=+719.428857297" watchObservedRunningTime="2026-07-24 03:29:29.990330437 +0000 UTC m=+719.429749904" Jul 24 03:29:36.006575 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:36.006538 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4f8cw" Jul 24 03:29:36.342213 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:36.342132 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:29:37.095960 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:37.095911 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:29:44.517972 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:44.517939 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:29:44.539759 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:44.539716 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5cfc5749b4-xnztl"] Jul 24 03:29:44.618422 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:44.618337 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:29:44.698345 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:44.698313 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cfc5749b4-xnztl"] Jul 24 03:29:44.698525 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:44.698407 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cfc5749b4-xnztl" Jul 24 03:29:44.703148 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:44.703125 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 03:29:44.703281 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:44.703252 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 03:29:44.703281 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:44.703266 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-j56qj\"" Jul 24 03:29:44.724377 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:44.724338 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:29:44.725668 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:44.725647 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-687994cdfd-tlnsg"] Jul 24 03:29:44.739604 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:44.739584 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-687994cdfd-tlnsg"] Jul 24 03:29:44.739703 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:44.739672 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-687994cdfd-tlnsg" Jul 24 03:29:44.741438 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:44.741413 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0fe0c5ff-e206-4d98-8fb2-b1d8762916f7-webhook-cert\") pod \"maas-controller-5cfc5749b4-xnztl\" (UID: \"0fe0c5ff-e206-4d98-8fb2-b1d8762916f7\") " pod="opendatahub/maas-controller-5cfc5749b4-xnztl" Jul 24 03:29:44.741587 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:44.741503 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vjnj\" (UniqueName: \"kubernetes.io/projected/0fe0c5ff-e206-4d98-8fb2-b1d8762916f7-kube-api-access-8vjnj\") pod \"maas-controller-5cfc5749b4-xnztl\" (UID: \"0fe0c5ff-e206-4d98-8fb2-b1d8762916f7\") " pod="opendatahub/maas-controller-5cfc5749b4-xnztl" Jul 24 03:29:44.809573 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:44.809499 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:29:44.842586 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:44.842550 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jhms\" (UniqueName: \"kubernetes.io/projected/ce3547aa-04d5-474a-a383-35be3fa725f8-kube-api-access-6jhms\") pod \"maas-controller-687994cdfd-tlnsg\" (UID: \"ce3547aa-04d5-474a-a383-35be3fa725f8\") " pod="opendatahub/maas-controller-687994cdfd-tlnsg" Jul 24 03:29:44.842851 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:44.842617 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0fe0c5ff-e206-4d98-8fb2-b1d8762916f7-webhook-cert\") pod \"maas-controller-5cfc5749b4-xnztl\" (UID: \"0fe0c5ff-e206-4d98-8fb2-b1d8762916f7\") " pod="opendatahub/maas-controller-5cfc5749b4-xnztl" Jul 24 03:29:44.842851 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:44.842649 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce3547aa-04d5-474a-a383-35be3fa725f8-webhook-cert\") pod \"maas-controller-687994cdfd-tlnsg\" (UID: \"ce3547aa-04d5-474a-a383-35be3fa725f8\") " pod="opendatahub/maas-controller-687994cdfd-tlnsg" Jul 24 03:29:44.842851 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:44.842778 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vjnj\" (UniqueName: \"kubernetes.io/projected/0fe0c5ff-e206-4d98-8fb2-b1d8762916f7-kube-api-access-8vjnj\") pod \"maas-controller-5cfc5749b4-xnztl\" (UID: \"0fe0c5ff-e206-4d98-8fb2-b1d8762916f7\") " pod="opendatahub/maas-controller-5cfc5749b4-xnztl" Jul 24 03:29:44.845160 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:44.845140 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0fe0c5ff-e206-4d98-8fb2-b1d8762916f7-webhook-cert\") pod \"maas-controller-5cfc5749b4-xnztl\" (UID: \"0fe0c5ff-e206-4d98-8fb2-b1d8762916f7\") " pod="opendatahub/maas-controller-5cfc5749b4-xnztl" Jul 24 03:29:44.851218 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:44.851197 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vjnj\" (UniqueName: \"kubernetes.io/projected/0fe0c5ff-e206-4d98-8fb2-b1d8762916f7-kube-api-access-8vjnj\") pod \"maas-controller-5cfc5749b4-xnztl\" (UID: \"0fe0c5ff-e206-4d98-8fb2-b1d8762916f7\") " pod="opendatahub/maas-controller-5cfc5749b4-xnztl" Jul 24 03:29:44.943815 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:44.943777 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jhms\" (UniqueName: \"kubernetes.io/projected/ce3547aa-04d5-474a-a383-35be3fa725f8-kube-api-access-6jhms\") pod \"maas-controller-687994cdfd-tlnsg\" (UID: \"ce3547aa-04d5-474a-a383-35be3fa725f8\") " pod="opendatahub/maas-controller-687994cdfd-tlnsg" Jul 24 03:29:44.943991 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:44.943832 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce3547aa-04d5-474a-a383-35be3fa725f8-webhook-cert\") pod \"maas-controller-687994cdfd-tlnsg\" (UID: \"ce3547aa-04d5-474a-a383-35be3fa725f8\") " pod="opendatahub/maas-controller-687994cdfd-tlnsg" Jul 24 03:29:44.946362 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:44.946334 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce3547aa-04d5-474a-a383-35be3fa725f8-webhook-cert\") pod \"maas-controller-687994cdfd-tlnsg\" (UID: \"ce3547aa-04d5-474a-a383-35be3fa725f8\") " pod="opendatahub/maas-controller-687994cdfd-tlnsg" Jul 24 03:29:44.952274 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:44.952253 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jhms\" (UniqueName: \"kubernetes.io/projected/ce3547aa-04d5-474a-a383-35be3fa725f8-kube-api-access-6jhms\") pod \"maas-controller-687994cdfd-tlnsg\" (UID: \"ce3547aa-04d5-474a-a383-35be3fa725f8\") " pod="opendatahub/maas-controller-687994cdfd-tlnsg" Jul 24 03:29:45.009035 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:45.009007 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cfc5749b4-xnztl" Jul 24 03:29:45.050437 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:45.050414 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-687994cdfd-tlnsg" Jul 24 03:29:45.168330 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:45.168299 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cfc5749b4-xnztl"] Jul 24 03:29:45.172867 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:29:45.172834 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe0c5ff_e206_4d98_8fb2_b1d8762916f7.slice/crio-e344ea8f489d73e6c53eaaf9d902e00d1db6b80ed025e93771c6f4969d4146ac WatchSource:0}: Error finding container e344ea8f489d73e6c53eaaf9d902e00d1db6b80ed025e93771c6f4969d4146ac: Status 404 returned error can't find the container with id e344ea8f489d73e6c53eaaf9d902e00d1db6b80ed025e93771c6f4969d4146ac Jul 24 03:29:45.202031 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:45.202002 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-687994cdfd-tlnsg"] Jul 24 03:29:45.205439 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:29:45.205414 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3547aa_04d5_474a_a383_35be3fa725f8.slice/crio-9b161ce0e22f0aa578217a79d827ff1124e691e41ceef6d06aacab65fec083d8 WatchSource:0}: Error finding container 9b161ce0e22f0aa578217a79d827ff1124e691e41ceef6d06aacab65fec083d8: Status 404 returned error can't find the container with id 9b161ce0e22f0aa578217a79d827ff1124e691e41ceef6d06aacab65fec083d8 Jul 24 03:29:46.059033 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:46.058951 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cfc5749b4-xnztl" event={"ID":"0fe0c5ff-e206-4d98-8fb2-b1d8762916f7","Type":"ContainerStarted","Data":"e344ea8f489d73e6c53eaaf9d902e00d1db6b80ed025e93771c6f4969d4146ac"} Jul 24 03:29:46.060939 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:46.060906 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-687994cdfd-tlnsg" event={"ID":"ce3547aa-04d5-474a-a383-35be3fa725f8","Type":"ContainerStarted","Data":"9b161ce0e22f0aa578217a79d827ff1124e691e41ceef6d06aacab65fec083d8"} Jul 24 03:29:47.624756 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:47.624710 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:29:47.725263 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:47.725222 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:29:47.825371 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:47.825333 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:29:47.925417 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:47.925333 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:29:48.075172 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:48.075133 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cfc5749b4-xnztl" event={"ID":"0fe0c5ff-e206-4d98-8fb2-b1d8762916f7","Type":"ContainerStarted","Data":"b6e1834450a1f073811c90574036a402bacfbe3801bf62d8c08df83b0070be18"} Jul 24 03:29:48.075361 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:48.075229 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5cfc5749b4-xnztl" Jul 24 03:29:48.076740 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:48.076718 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-687994cdfd-tlnsg" event={"ID":"ce3547aa-04d5-474a-a383-35be3fa725f8","Type":"ContainerStarted","Data":"2f0da35572a3bb566fcf2415b584bd9a26f8f30a4bbac1692ea0eff613ee250a"} Jul 24 03:29:48.076855 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:48.076840 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-687994cdfd-tlnsg" Jul 24 03:29:48.093994 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:48.093956 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5cfc5749b4-xnztl" podStartSLOduration=1.631578015 podStartE2EDuration="4.093941992s" totalImagesPullingTime="2.462363977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:29:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:29:48.092586912 +0000 UTC m=+737.532006372" watchObservedRunningTime="2026-07-24 03:29:48.093941992 +0000 UTC m=+737.533361452" Jul 24 03:29:48.117382 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:48.117330 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-687994cdfd-tlnsg" podStartSLOduration=1.6946414440000002 podStartE2EDuration="4.117314499s" totalImagesPullingTime="2.422673055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:29:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:29:48.108780726 +0000 UTC m=+737.548200186" watchObservedRunningTime="2026-07-24 03:29:48.117314499 +0000 UTC m=+737.556733964" Jul 24 03:29:59.084508 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:59.084448 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5cfc5749b4-xnztl" Jul 24 03:29:59.084981 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:59.084554 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-687994cdfd-tlnsg" Jul 24 03:29:59.153981 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:59.153944 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cfc5749b4-xnztl"] Jul 24 03:29:59.154189 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:59.154165 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5cfc5749b4-xnztl" podUID="0fe0c5ff-e206-4d98-8fb2-b1d8762916f7" containerName="manager" containerID="cri-o://b6e1834450a1f073811c90574036a402bacfbe3801bf62d8c08df83b0070be18" gracePeriod=30 Jul 24 03:29:59.407541 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:59.407514 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cfc5749b4-xnztl" Jul 24 03:29:59.480456 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:59.480429 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0fe0c5ff-e206-4d98-8fb2-b1d8762916f7-webhook-cert\") pod \"0fe0c5ff-e206-4d98-8fb2-b1d8762916f7\" (UID: \"0fe0c5ff-e206-4d98-8fb2-b1d8762916f7\") " Jul 24 03:29:59.480639 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:59.480527 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vjnj\" (UniqueName: \"kubernetes.io/projected/0fe0c5ff-e206-4d98-8fb2-b1d8762916f7-kube-api-access-8vjnj\") pod \"0fe0c5ff-e206-4d98-8fb2-b1d8762916f7\" (UID: \"0fe0c5ff-e206-4d98-8fb2-b1d8762916f7\") " Jul 24 03:29:59.482722 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:59.482686 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fe0c5ff-e206-4d98-8fb2-b1d8762916f7-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0fe0c5ff-e206-4d98-8fb2-b1d8762916f7" (UID: "0fe0c5ff-e206-4d98-8fb2-b1d8762916f7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 03:29:59.482830 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:59.482806 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe0c5ff-e206-4d98-8fb2-b1d8762916f7-kube-api-access-8vjnj" (OuterVolumeSpecName: "kube-api-access-8vjnj") pod "0fe0c5ff-e206-4d98-8fb2-b1d8762916f7" (UID: "0fe0c5ff-e206-4d98-8fb2-b1d8762916f7"). InnerVolumeSpecName "kube-api-access-8vjnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:29:59.581603 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:59.581574 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vjnj\" (UniqueName: \"kubernetes.io/projected/0fe0c5ff-e206-4d98-8fb2-b1d8762916f7-kube-api-access-8vjnj\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:29:59.581603 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:29:59.581600 2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0fe0c5ff-e206-4d98-8fb2-b1d8762916f7-webhook-cert\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:30:00.136499 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:00.136442 2579 generic.go:356] "Generic (PLEG): container finished" podID="0fe0c5ff-e206-4d98-8fb2-b1d8762916f7" containerID="b6e1834450a1f073811c90574036a402bacfbe3801bf62d8c08df83b0070be18" exitCode=0 Jul 24 03:30:00.136930 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:00.136515 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cfc5749b4-xnztl" Jul 24 03:30:00.136930 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:00.136528 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cfc5749b4-xnztl" event={"ID":"0fe0c5ff-e206-4d98-8fb2-b1d8762916f7","Type":"ContainerDied","Data":"b6e1834450a1f073811c90574036a402bacfbe3801bf62d8c08df83b0070be18"} Jul 24 03:30:00.136930 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:00.136580 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cfc5749b4-xnztl" event={"ID":"0fe0c5ff-e206-4d98-8fb2-b1d8762916f7","Type":"ContainerDied","Data":"e344ea8f489d73e6c53eaaf9d902e00d1db6b80ed025e93771c6f4969d4146ac"} Jul 24 03:30:00.136930 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:00.136601 2579 scope.go:117] "RemoveContainer" containerID="b6e1834450a1f073811c90574036a402bacfbe3801bf62d8c08df83b0070be18" Jul 24 03:30:00.145790 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:00.145776 2579 scope.go:117] "RemoveContainer" containerID="b6e1834450a1f073811c90574036a402bacfbe3801bf62d8c08df83b0070be18" Jul 24 03:30:00.146024 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:30:00.146007 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6e1834450a1f073811c90574036a402bacfbe3801bf62d8c08df83b0070be18\": container with ID starting with b6e1834450a1f073811c90574036a402bacfbe3801bf62d8c08df83b0070be18 not found: ID does not exist" containerID="b6e1834450a1f073811c90574036a402bacfbe3801bf62d8c08df83b0070be18" Jul 24 03:30:00.146072 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:00.146031 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e1834450a1f073811c90574036a402bacfbe3801bf62d8c08df83b0070be18"} err="failed to get container status \"b6e1834450a1f073811c90574036a402bacfbe3801bf62d8c08df83b0070be18\": rpc error: code = NotFound desc = could not find container \"b6e1834450a1f073811c90574036a402bacfbe3801bf62d8c08df83b0070be18\": container with ID starting with b6e1834450a1f073811c90574036a402bacfbe3801bf62d8c08df83b0070be18 not found: ID does not exist" Jul 24 03:30:00.157175 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:00.157153 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cfc5749b4-xnztl"] Jul 24 03:30:00.161317 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:00.161298 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5cfc5749b4-xnztl"] Jul 24 03:30:01.033309 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:01.033279 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fe0c5ff-e206-4d98-8fb2-b1d8762916f7" path="/var/lib/kubelet/pods/0fe0c5ff-e206-4d98-8fb2-b1d8762916f7/volumes" Jul 24 03:30:01.710280 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:01.710246 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:30:18.493250 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:18.493211 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:30:18.590961 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:18.590929 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:30:19.094145 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.094103 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:30:19.187864 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.187833 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:30:19.200680 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.200650 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-857c5484bd-xlxc2"] Jul 24 03:30:19.201029 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.201017 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fe0c5ff-e206-4d98-8fb2-b1d8762916f7" containerName="manager" Jul 24 03:30:19.201078 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.201031 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fe0c5ff-e206-4d98-8fb2-b1d8762916f7" containerName="manager" Jul 24 03:30:19.201114 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.201104 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fe0c5ff-e206-4d98-8fb2-b1d8762916f7" containerName="manager" Jul 24 03:30:19.205576 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.205548 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-857c5484bd-xlxc2" Jul 24 03:30:19.208563 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.208540 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 24 03:30:19.208667 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.208540 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-58nst\"" Jul 24 03:30:19.213786 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.213763 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-857c5484bd-xlxc2"] Jul 24 03:30:19.359322 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.359294 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/810d712e-8532-4bc5-a2a4-12b2b2f37330-openshift-service-ca\") pod \"maas-api-857c5484bd-xlxc2\" (UID: \"810d712e-8532-4bc5-a2a4-12b2b2f37330\") " pod="odh-ai-gateway-infra/maas-api-857c5484bd-xlxc2" Jul 24 03:30:19.359560 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.359339 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/810d712e-8532-4bc5-a2a4-12b2b2f37330-maas-api-tls\") pod \"maas-api-857c5484bd-xlxc2\" (UID: \"810d712e-8532-4bc5-a2a4-12b2b2f37330\") " pod="odh-ai-gateway-infra/maas-api-857c5484bd-xlxc2" Jul 24 03:30:19.359560 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.359417 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdlwf\" (UniqueName: \"kubernetes.io/projected/810d712e-8532-4bc5-a2a4-12b2b2f37330-kube-api-access-xdlwf\") pod \"maas-api-857c5484bd-xlxc2\" (UID: \"810d712e-8532-4bc5-a2a4-12b2b2f37330\") " pod="odh-ai-gateway-infra/maas-api-857c5484bd-xlxc2" Jul 24 03:30:19.460670 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.460639 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdlwf\" (UniqueName: \"kubernetes.io/projected/810d712e-8532-4bc5-a2a4-12b2b2f37330-kube-api-access-xdlwf\") pod \"maas-api-857c5484bd-xlxc2\" (UID: \"810d712e-8532-4bc5-a2a4-12b2b2f37330\") " pod="odh-ai-gateway-infra/maas-api-857c5484bd-xlxc2" Jul 24 03:30:19.460825 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.460700 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/810d712e-8532-4bc5-a2a4-12b2b2f37330-openshift-service-ca\") pod \"maas-api-857c5484bd-xlxc2\" (UID: \"810d712e-8532-4bc5-a2a4-12b2b2f37330\") " pod="odh-ai-gateway-infra/maas-api-857c5484bd-xlxc2" Jul 24 03:30:19.460825 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.460732 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/810d712e-8532-4bc5-a2a4-12b2b2f37330-maas-api-tls\") pod \"maas-api-857c5484bd-xlxc2\" (UID: \"810d712e-8532-4bc5-a2a4-12b2b2f37330\") " pod="odh-ai-gateway-infra/maas-api-857c5484bd-xlxc2" Jul 24 03:30:19.461384 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.461363 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/810d712e-8532-4bc5-a2a4-12b2b2f37330-openshift-service-ca\") pod \"maas-api-857c5484bd-xlxc2\" (UID: \"810d712e-8532-4bc5-a2a4-12b2b2f37330\") " pod="odh-ai-gateway-infra/maas-api-857c5484bd-xlxc2" Jul 24 03:30:19.463177 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.463160 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/810d712e-8532-4bc5-a2a4-12b2b2f37330-maas-api-tls\") pod \"maas-api-857c5484bd-xlxc2\" (UID: \"810d712e-8532-4bc5-a2a4-12b2b2f37330\") " pod="odh-ai-gateway-infra/maas-api-857c5484bd-xlxc2" Jul 24 03:30:19.468854 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.468818 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdlwf\" (UniqueName: \"kubernetes.io/projected/810d712e-8532-4bc5-a2a4-12b2b2f37330-kube-api-access-xdlwf\") pod \"maas-api-857c5484bd-xlxc2\" (UID: \"810d712e-8532-4bc5-a2a4-12b2b2f37330\") " pod="odh-ai-gateway-infra/maas-api-857c5484bd-xlxc2" Jul 24 03:30:19.516936 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.516910 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-857c5484bd-xlxc2" Jul 24 03:30:19.542595 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.542566 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:30:19.640502 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.640018 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:30:19.700274 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.700244 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-857c5484bd-xlxc2"] Jul 24 03:30:19.702667 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:30:19.702640 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod810d712e_8532_4bc5_a2a4_12b2b2f37330.slice/crio-f85bd296f3c84a007c61456be20bbea7416f085a591713c40367df93fc495545 WatchSource:0}: Error finding container f85bd296f3c84a007c61456be20bbea7416f085a591713c40367df93fc495545: Status 404 returned error can't find the container with id f85bd296f3c84a007c61456be20bbea7416f085a591713c40367df93fc495545 Jul 24 03:30:19.726187 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.726154 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-fczls"] Jul 24 03:30:19.730853 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.730835 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-fczls" Jul 24 03:30:19.733800 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.733780 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-nzcpz\"" Jul 24 03:30:19.733984 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.733963 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 24 03:30:19.745763 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.745696 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-fczls"] Jul 24 03:30:19.865061 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.865025 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmjqq\" (UniqueName: \"kubernetes.io/projected/5e37496f-fe2b-4acb-97df-faefd815ca0d-kube-api-access-kmjqq\") pod \"payload-pre-processing-687dbb6cf6-fczls\" (UID: \"5e37496f-fe2b-4acb-97df-faefd815ca0d\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-fczls" Jul 24 03:30:19.865280 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.865092 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e37496f-fe2b-4acb-97df-faefd815ca0d-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-fczls\" (UID: \"5e37496f-fe2b-4acb-97df-faefd815ca0d\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-fczls" Jul 24 03:30:19.966433 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.966352 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmjqq\" (UniqueName: \"kubernetes.io/projected/5e37496f-fe2b-4acb-97df-faefd815ca0d-kube-api-access-kmjqq\") pod \"payload-pre-processing-687dbb6cf6-fczls\" (UID: \"5e37496f-fe2b-4acb-97df-faefd815ca0d\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-fczls" Jul 24 03:30:19.966433 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.966426 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e37496f-fe2b-4acb-97df-faefd815ca0d-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-fczls\" (UID: \"5e37496f-fe2b-4acb-97df-faefd815ca0d\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-fczls" Jul 24 03:30:19.967156 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.967135 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e37496f-fe2b-4acb-97df-faefd815ca0d-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-fczls\" (UID: \"5e37496f-fe2b-4acb-97df-faefd815ca0d\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-fczls" Jul 24 03:30:19.975883 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:19.975857 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmjqq\" (UniqueName: \"kubernetes.io/projected/5e37496f-fe2b-4acb-97df-faefd815ca0d-kube-api-access-kmjqq\") pod \"payload-pre-processing-687dbb6cf6-fczls\" (UID: \"5e37496f-fe2b-4acb-97df-faefd815ca0d\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-fczls" Jul 24 03:30:20.041714 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:20.041676 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-fczls" Jul 24 03:30:20.234586 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:20.234502 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-857c5484bd-xlxc2" event={"ID":"810d712e-8532-4bc5-a2a4-12b2b2f37330","Type":"ContainerStarted","Data":"f85bd296f3c84a007c61456be20bbea7416f085a591713c40367df93fc495545"} Jul 24 03:30:20.410825 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:20.410798 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-fczls"] Jul 24 03:30:20.413327 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:30:20.413296 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e37496f_fe2b_4acb_97df_faefd815ca0d.slice/crio-e90fdd6e8f3f298191c8349b0a0231d72f5924e61f89823dcdfc568a69a767ad WatchSource:0}: Error finding container e90fdd6e8f3f298191c8349b0a0231d72f5924e61f89823dcdfc568a69a767ad: Status 404 returned error can't find the container with id e90fdd6e8f3f298191c8349b0a0231d72f5924e61f89823dcdfc568a69a767ad Jul 24 03:30:21.244335 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:21.244297 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-fczls" event={"ID":"5e37496f-fe2b-4acb-97df-faefd815ca0d","Type":"ContainerStarted","Data":"e90fdd6e8f3f298191c8349b0a0231d72f5924e61f89823dcdfc568a69a767ad"} Jul 24 03:30:22.251834 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:22.251791 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-857c5484bd-xlxc2" event={"ID":"810d712e-8532-4bc5-a2a4-12b2b2f37330","Type":"ContainerStarted","Data":"d5bdec6bd280706073f3ead0a842107e548b1cdd98ad3a01e4a8d410d3b55e5c"} Jul 24 03:30:22.252349 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:22.251884 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-857c5484bd-xlxc2" Jul 24 03:30:22.268644 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:22.268577 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-857c5484bd-xlxc2" podStartSLOduration=1.258257399 podStartE2EDuration="3.268558156s" totalImagesPullingTime="2.010300757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:30:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:30:22.267779658 +0000 UTC m=+771.707199120" watchObservedRunningTime="2026-07-24 03:30:22.268558156 +0000 UTC m=+771.707977615" Jul 24 03:30:24.262993 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:24.262958 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-fczls" event={"ID":"5e37496f-fe2b-4acb-97df-faefd815ca0d","Type":"ContainerStarted","Data":"710e8eb20b182f63f10c63a6eab4273f6a8c60dbdefe7c01eef77b227cca9cad"} Jul 24 03:30:24.263362 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:24.263087 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-fczls" Jul 24 03:30:24.286217 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:24.286140 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-fczls" podStartSLOduration=2.372446068 podStartE2EDuration="5.286121481s" totalImagesPullingTime="2.913675413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:30:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:30:24.283048337 +0000 UTC m=+773.722467798" watchObservedRunningTime="2026-07-24 03:30:24.286121481 +0000 UTC m=+773.725540944" Jul 24 03:30:27.278704 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:27.278667 2579 generic.go:356] "Generic (PLEG): container finished" podID="e6304eb4-8cda-4f2b-8d31-b2da9c27a10a" containerID="d7083a9f0ec86e5f5a212f236c1979d12f21e585558416bf50cfe52f6bab4683" exitCode=0 Jul 24 03:30:27.279082 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:27.278739 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" event={"ID":"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a","Type":"ContainerDied","Data":"d7083a9f0ec86e5f5a212f236c1979d12f21e585558416bf50cfe52f6bab4683"} Jul 24 03:30:27.279218 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:27.279202 2579 scope.go:117] "RemoveContainer" containerID="d7083a9f0ec86e5f5a212f236c1979d12f21e585558416bf50cfe52f6bab4683" Jul 24 03:30:27.280212 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:27.280186 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:30:27.280282 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:27.280269 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:30:27.280328 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:27.280316 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:30:28.261959 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:28.261933 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-857c5484bd-xlxc2" Jul 24 03:30:28.290312 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:28.290273 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" event={"ID":"e6304eb4-8cda-4f2b-8d31-b2da9c27a10a","Type":"ContainerStarted","Data":"377b41a73a6315e5f9966abee9c7a695c941c2b9830e104406831fc8e21f6a04"} Jul 24 03:30:28.513168 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:28.513073 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:30:28.513336 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:28.513297 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-8s6w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 24 03:30:28.513405 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:28.513357 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" podUID="e6304eb4-8cda-4f2b-8d31-b2da9c27a10a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 24 03:30:29.307318 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:29.307282 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:30:29.410110 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:29.410078 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:30:29.513050 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:29.513017 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-8s6w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 24 03:30:29.513216 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:29.513084 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" podUID="e6304eb4-8cda-4f2b-8d31-b2da9c27a10a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 24 03:30:30.513627 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:30.513583 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-8s6w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 24 03:30:30.514043 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:30.513649 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" podUID="e6304eb4-8cda-4f2b-8d31-b2da9c27a10a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 24 03:30:31.512858 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:31.512817 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-8s6w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 24 03:30:31.513047 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:31.512877 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" podUID="e6304eb4-8cda-4f2b-8d31-b2da9c27a10a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 24 03:30:32.516394 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:32.516363 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:30:32.516795 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:32.516704 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:30:32.517364 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:32.517344 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8s6w4" Jul 24 03:30:35.270930 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:35.270893 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-fczls" Jul 24 03:30:42.021446 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:42.021413 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj"] Jul 24 03:30:42.025082 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:42.025061 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj" Jul 24 03:30:42.027454 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:42.027434 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 24 03:30:42.027556 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:42.027436 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 24 03:30:42.028389 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:42.028373 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-p7qhl\"" Jul 24 03:30:42.028491 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:42.028420 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 24 03:30:42.033818 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:42.033794 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj"] Jul 24 03:30:42.053746 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:42.053721 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:30:42.110097 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:42.110049 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30343da6-f4a4-4bdc-be6c-f4978d409439-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj\" (UID: \"30343da6-f4a4-4bdc-be6c-f4978d409439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj" Jul 24 03:30:42.110268 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:42.110114 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30343da6-f4a4-4bdc-be6c-f4978d409439-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj\" (UID: \"30343da6-f4a4-4bdc-be6c-f4978d409439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj" Jul 24 03:30:42.110339 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:42.110259 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30343da6-f4a4-4bdc-be6c-f4978d409439-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj\" (UID: \"30343da6-f4a4-4bdc-be6c-f4978d409439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj" Jul 24 03:30:42.110339 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:42.110300 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30343da6-f4a4-4bdc-be6c-f4978d409439-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj\" (UID: \"30343da6-f4a4-4bdc-be6c-f4978d409439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj" Jul 24 03:30:42.110455 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:42.110388 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf6ck\" (UniqueName: \"kubernetes.io/projected/30343da6-f4a4-4bdc-be6c-f4978d409439-kube-api-access-pf6ck\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj\" (UID: \"30343da6-f4a4-4bdc-be6c-f4978d409439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj" Jul 24 03:30:42.110544 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:42.110454 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30343da6-f4a4-4bdc-be6c-f4978d409439-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj\" (UID: \"30343da6-f4a4-4bdc-be6c-f4978d409439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj" Jul 24 03:30:42.211797 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:42.211767 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30343da6-f4a4-4bdc-be6c-f4978d409439-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj\" (UID: \"30343da6-f4a4-4bdc-be6c-f4978d409439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj" Jul 24 03:30:42.211990 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:42.211819 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30343da6-f4a4-4bdc-be6c-f4978d409439-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj\" (UID: \"30343da6-f4a4-4bdc-be6c-f4978d409439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj" Jul 24 03:30:42.211990 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:42.211846 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30343da6-f4a4-4bdc-be6c-f4978d409439-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj\" (UID: \"30343da6-f4a4-4bdc-be6c-f4978d409439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj" Jul 24 03:30:42.211990 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:42.211895 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30343da6-f4a4-4bdc-be6c-f4978d409439-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj\" (UID: \"30343da6-f4a4-4bdc-be6c-f4978d409439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj" Jul 24 03:30:42.211990 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:42.211917 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30343da6-f4a4-4bdc-be6c-f4978d409439-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj\" (UID: \"30343da6-f4a4-4bdc-be6c-f4978d409439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj" Jul 24 03:30:42.211990 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:42.211956 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf6ck\" (UniqueName: \"kubernetes.io/projected/30343da6-f4a4-4bdc-be6c-f4978d409439-kube-api-access-pf6ck\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj\" (UID: \"30343da6-f4a4-4bdc-be6c-f4978d409439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj" Jul 24 03:30:42.212247 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:42.212214 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30343da6-f4a4-4bdc-be6c-f4978d409439-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj\" (UID: \"30343da6-f4a4-4bdc-be6c-f4978d409439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj" Jul 24 03:30:42.212307 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:42.212285 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30343da6-f4a4-4bdc-be6c-f4978d409439-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj\" (UID: \"30343da6-f4a4-4bdc-be6c-f4978d409439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj" Jul 24 03:30:42.212355 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:42.212328 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30343da6-f4a4-4bdc-be6c-f4978d409439-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj\" (UID: \"30343da6-f4a4-4bdc-be6c-f4978d409439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj" Jul 24 03:30:42.214435 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:42.214412 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30343da6-f4a4-4bdc-be6c-f4978d409439-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj\" (UID: \"30343da6-f4a4-4bdc-be6c-f4978d409439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj" Jul 24 03:30:42.214620 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:42.214602 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30343da6-f4a4-4bdc-be6c-f4978d409439-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj\" (UID: \"30343da6-f4a4-4bdc-be6c-f4978d409439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj" Jul 24 03:30:42.220110 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:42.220092 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf6ck\" (UniqueName: \"kubernetes.io/projected/30343da6-f4a4-4bdc-be6c-f4978d409439-kube-api-access-pf6ck\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj\" (UID: \"30343da6-f4a4-4bdc-be6c-f4978d409439\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj" Jul 24 03:30:42.337231 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:42.337164 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj" Jul 24 03:30:42.471841 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:42.471817 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj"] Jul 24 03:30:42.474086 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:30:42.474053 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30343da6_f4a4_4bdc_be6c_f4978d409439.slice/crio-73c934929e754b4d064e49be427fd3ae51f28252c12cd3b6bf7861431b386bc1 WatchSource:0}: Error finding container 73c934929e754b4d064e49be427fd3ae51f28252c12cd3b6bf7861431b386bc1: Status 404 returned error can't find the container with id 73c934929e754b4d064e49be427fd3ae51f28252c12cd3b6bf7861431b386bc1 Jul 24 03:30:43.370100 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:43.370066 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj" event={"ID":"30343da6-f4a4-4bdc-be6c-f4978d409439","Type":"ContainerStarted","Data":"73c934929e754b4d064e49be427fd3ae51f28252c12cd3b6bf7861431b386bc1"} Jul 24 03:30:44.376357 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:44.376319 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj" event={"ID":"30343da6-f4a4-4bdc-be6c-f4978d409439","Type":"ContainerStarted","Data":"7d26e4dac1c6c1ef65b2b8515151ecb2a98245f84171297367fd96606f273687"} Jul 24 03:30:44.400723 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:44.400672 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj" podStartSLOduration=1.3502000650000001 podStartE2EDuration="2.400656842s" totalImagesPullingTime="1.050456777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:30:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:30:44.399507481 +0000 UTC m=+793.838926943" watchObservedRunningTime="2026-07-24 03:30:44.400656842 +0000 UTC m=+793.840076301" Jul 24 03:30:51.660234 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:51.660204 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:30:52.338103 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:52.338066 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj" Jul 24 03:30:52.338296 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:52.338129 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj" Jul 24 03:30:52.350882 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:52.350856 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj" Jul 24 03:30:52.430995 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:52.430967 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj" Jul 24 03:30:57.873326 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:57.873293 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf"] Jul 24 03:30:57.879277 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:57.879253 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf" Jul 24 03:30:57.882687 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:57.882661 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 24 03:30:57.900647 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:57.900623 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf"] Jul 24 03:30:57.914387 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:57.914363 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:30:57.952744 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:57.952705 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2078c0a8-340d-471d-8d2c-a022813c0ef9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cvztf\" (UID: \"2078c0a8-340d-471d-8d2c-a022813c0ef9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf" Jul 24 03:30:57.952880 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:57.952748 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2078c0a8-340d-471d-8d2c-a022813c0ef9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cvztf\" (UID: \"2078c0a8-340d-471d-8d2c-a022813c0ef9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf" Jul 24 03:30:57.952880 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:57.952778 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2078c0a8-340d-471d-8d2c-a022813c0ef9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cvztf\" (UID: \"2078c0a8-340d-471d-8d2c-a022813c0ef9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf" Jul 24 03:30:57.952880 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:57.952821 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2078c0a8-340d-471d-8d2c-a022813c0ef9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cvztf\" (UID: \"2078c0a8-340d-471d-8d2c-a022813c0ef9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf" Jul 24 03:30:57.952880 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:57.952847 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2078c0a8-340d-471d-8d2c-a022813c0ef9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cvztf\" (UID: \"2078c0a8-340d-471d-8d2c-a022813c0ef9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf" Jul 24 03:30:57.953017 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:57.952882 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxt27\" (UniqueName: \"kubernetes.io/projected/2078c0a8-340d-471d-8d2c-a022813c0ef9-kube-api-access-rxt27\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cvztf\" (UID: \"2078c0a8-340d-471d-8d2c-a022813c0ef9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf" Jul 24 03:30:58.054032 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:58.053995 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2078c0a8-340d-471d-8d2c-a022813c0ef9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cvztf\" (UID: \"2078c0a8-340d-471d-8d2c-a022813c0ef9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf" Jul 24 03:30:58.054032 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:58.054032 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2078c0a8-340d-471d-8d2c-a022813c0ef9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cvztf\" (UID: \"2078c0a8-340d-471d-8d2c-a022813c0ef9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf" Jul 24 03:30:58.054742 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:58.054068 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxt27\" (UniqueName: \"kubernetes.io/projected/2078c0a8-340d-471d-8d2c-a022813c0ef9-kube-api-access-rxt27\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cvztf\" (UID: \"2078c0a8-340d-471d-8d2c-a022813c0ef9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf" Jul 24 03:30:58.054742 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:58.054113 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2078c0a8-340d-471d-8d2c-a022813c0ef9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cvztf\" (UID: \"2078c0a8-340d-471d-8d2c-a022813c0ef9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf" Jul 24 03:30:58.054742 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:58.054142 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2078c0a8-340d-471d-8d2c-a022813c0ef9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cvztf\" (UID: \"2078c0a8-340d-471d-8d2c-a022813c0ef9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf" Jul 24 03:30:58.054742 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:58.054176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2078c0a8-340d-471d-8d2c-a022813c0ef9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cvztf\" (UID: \"2078c0a8-340d-471d-8d2c-a022813c0ef9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf" Jul 24 03:30:58.054742 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:58.054441 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2078c0a8-340d-471d-8d2c-a022813c0ef9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cvztf\" (UID: \"2078c0a8-340d-471d-8d2c-a022813c0ef9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf" Jul 24 03:30:58.054742 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:58.054572 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2078c0a8-340d-471d-8d2c-a022813c0ef9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cvztf\" (UID: \"2078c0a8-340d-471d-8d2c-a022813c0ef9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf" Jul 24 03:30:58.054742 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:58.054623 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2078c0a8-340d-471d-8d2c-a022813c0ef9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cvztf\" (UID: \"2078c0a8-340d-471d-8d2c-a022813c0ef9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf" Jul 24 03:30:58.056488 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:58.056451 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2078c0a8-340d-471d-8d2c-a022813c0ef9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cvztf\" (UID: \"2078c0a8-340d-471d-8d2c-a022813c0ef9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf" Jul 24 03:30:58.056826 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:58.056809 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2078c0a8-340d-471d-8d2c-a022813c0ef9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cvztf\" (UID: \"2078c0a8-340d-471d-8d2c-a022813c0ef9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf" Jul 24 03:30:58.062618 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:58.062598 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxt27\" (UniqueName: \"kubernetes.io/projected/2078c0a8-340d-471d-8d2c-a022813c0ef9-kube-api-access-rxt27\") pod \"e2e-trlp-test-simulated-kserve-598466548d-cvztf\" (UID: \"2078c0a8-340d-471d-8d2c-a022813c0ef9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf" Jul 24 03:30:58.190619 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:58.190530 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf" Jul 24 03:30:58.331533 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:58.331507 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf"] Jul 24 03:30:58.333160 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:30:58.333126 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2078c0a8_340d_471d_8d2c_a022813c0ef9.slice/crio-312e439dfcfc66722b2d08eac94ef583cd50bc28cbc195b1e18b97e1e55ae564 WatchSource:0}: Error finding container 312e439dfcfc66722b2d08eac94ef583cd50bc28cbc195b1e18b97e1e55ae564: Status 404 returned error can't find the container with id 312e439dfcfc66722b2d08eac94ef583cd50bc28cbc195b1e18b97e1e55ae564 Jul 24 03:30:58.449040 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:58.448951 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf" event={"ID":"2078c0a8-340d-471d-8d2c-a022813c0ef9","Type":"ContainerStarted","Data":"312e439dfcfc66722b2d08eac94ef583cd50bc28cbc195b1e18b97e1e55ae564"} Jul 24 03:30:59.455060 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:59.455016 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf" event={"ID":"2078c0a8-340d-471d-8d2c-a022813c0ef9","Type":"ContainerStarted","Data":"07590d474dff2768e061186d623aa27ab8de055258bfc47b6dd1ad1ed501b55c"} Jul 24 03:30:59.478721 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:30:59.478680 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf" podStartSLOduration=2.304611657 podStartE2EDuration="2.478666335s" totalImagesPullingTime="174.054678ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:30:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:30:59.477163984 +0000 UTC m=+808.916583443" watchObservedRunningTime="2026-07-24 03:30:59.478666335 +0000 UTC m=+808.918085836" Jul 24 03:31:06.462148 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:06.462120 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:31:08.191319 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:08.191279 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf" Jul 24 03:31:08.191757 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:08.191347 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf" Jul 24 03:31:08.203899 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:08.203876 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf" Jul 24 03:31:08.511487 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:08.511395 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-cvztf" Jul 24 03:31:30.664746 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:30.664710 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:31:42.722915 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:42.722881 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp"] Jul 24 03:31:42.727604 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:42.727578 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp" Jul 24 03:31:42.730146 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:42.730128 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 24 03:31:42.734748 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:42.734661 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp"] Jul 24 03:31:42.754038 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:42.754014 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:31:42.858895 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:42.858865 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/103d6fe6-8801-44bd-9b7a-f1a9ad93080d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp\" (UID: \"103d6fe6-8801-44bd-9b7a-f1a9ad93080d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp" Jul 24 03:31:42.859051 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:42.858931 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103d6fe6-8801-44bd-9b7a-f1a9ad93080d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp\" (UID: \"103d6fe6-8801-44bd-9b7a-f1a9ad93080d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp" Jul 24 03:31:42.859051 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:42.858958 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/103d6fe6-8801-44bd-9b7a-f1a9ad93080d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp\" (UID: \"103d6fe6-8801-44bd-9b7a-f1a9ad93080d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp" Jul 24 03:31:42.859051 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:42.858987 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/103d6fe6-8801-44bd-9b7a-f1a9ad93080d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp\" (UID: \"103d6fe6-8801-44bd-9b7a-f1a9ad93080d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp" Jul 24 03:31:42.859051 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:42.859015 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/103d6fe6-8801-44bd-9b7a-f1a9ad93080d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp\" (UID: \"103d6fe6-8801-44bd-9b7a-f1a9ad93080d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp" Jul 24 03:31:42.859190 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:42.859087 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vf5v\" (UniqueName: \"kubernetes.io/projected/103d6fe6-8801-44bd-9b7a-f1a9ad93080d-kube-api-access-6vf5v\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp\" (UID: \"103d6fe6-8801-44bd-9b7a-f1a9ad93080d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp" Jul 24 03:31:42.960167 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:42.960133 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/103d6fe6-8801-44bd-9b7a-f1a9ad93080d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp\" (UID: \"103d6fe6-8801-44bd-9b7a-f1a9ad93080d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp" Jul 24 03:31:42.960328 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:42.960181 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103d6fe6-8801-44bd-9b7a-f1a9ad93080d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp\" (UID: \"103d6fe6-8801-44bd-9b7a-f1a9ad93080d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp" Jul 24 03:31:42.960328 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:42.960210 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/103d6fe6-8801-44bd-9b7a-f1a9ad93080d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp\" (UID: \"103d6fe6-8801-44bd-9b7a-f1a9ad93080d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp" Jul 24 03:31:42.960328 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:42.960238 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/103d6fe6-8801-44bd-9b7a-f1a9ad93080d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp\" (UID: \"103d6fe6-8801-44bd-9b7a-f1a9ad93080d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp" Jul 24 03:31:42.960328 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:42.960263 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/103d6fe6-8801-44bd-9b7a-f1a9ad93080d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp\" (UID: \"103d6fe6-8801-44bd-9b7a-f1a9ad93080d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp" Jul 24 03:31:42.960328 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:42.960295 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vf5v\" (UniqueName: \"kubernetes.io/projected/103d6fe6-8801-44bd-9b7a-f1a9ad93080d-kube-api-access-6vf5v\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp\" (UID: \"103d6fe6-8801-44bd-9b7a-f1a9ad93080d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp" Jul 24 03:31:42.960629 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:42.960605 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/103d6fe6-8801-44bd-9b7a-f1a9ad93080d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp\" (UID: \"103d6fe6-8801-44bd-9b7a-f1a9ad93080d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp" Jul 24 03:31:42.960698 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:42.960641 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103d6fe6-8801-44bd-9b7a-f1a9ad93080d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp\" (UID: \"103d6fe6-8801-44bd-9b7a-f1a9ad93080d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp" Jul 24 03:31:42.960754 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:42.960724 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/103d6fe6-8801-44bd-9b7a-f1a9ad93080d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp\" (UID: \"103d6fe6-8801-44bd-9b7a-f1a9ad93080d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp" Jul 24 03:31:42.962693 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:42.962667 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/103d6fe6-8801-44bd-9b7a-f1a9ad93080d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp\" (UID: \"103d6fe6-8801-44bd-9b7a-f1a9ad93080d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp" Jul 24 03:31:42.962891 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:42.962876 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/103d6fe6-8801-44bd-9b7a-f1a9ad93080d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp\" (UID: \"103d6fe6-8801-44bd-9b7a-f1a9ad93080d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp" Jul 24 03:31:42.967533 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:42.967509 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vf5v\" (UniqueName: \"kubernetes.io/projected/103d6fe6-8801-44bd-9b7a-f1a9ad93080d-kube-api-access-6vf5v\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp\" (UID: \"103d6fe6-8801-44bd-9b7a-f1a9ad93080d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp" Jul 24 03:31:43.038167 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:43.038104 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp" Jul 24 03:31:43.173068 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:43.173046 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp"] Jul 24 03:31:43.175145 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:31:43.175102 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod103d6fe6_8801_44bd_9b7a_f1a9ad93080d.slice/crio-2de4602f6a373fcfb2ebef0bcc35486dda2fe896d96afb294c6b69c2ebfab4b8 WatchSource:0}: Error finding container 2de4602f6a373fcfb2ebef0bcc35486dda2fe896d96afb294c6b69c2ebfab4b8: Status 404 returned error can't find the container with id 2de4602f6a373fcfb2ebef0bcc35486dda2fe896d96afb294c6b69c2ebfab4b8 Jul 24 03:31:43.674659 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:43.674624 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp" event={"ID":"103d6fe6-8801-44bd-9b7a-f1a9ad93080d","Type":"ContainerStarted","Data":"f5640fc1e79809959710c82f308a27963a7c8d125dd96d183295e92ac1c7f3c0"} Jul 24 03:31:43.674659 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:43.674663 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp" event={"ID":"103d6fe6-8801-44bd-9b7a-f1a9ad93080d","Type":"ContainerStarted","Data":"2de4602f6a373fcfb2ebef0bcc35486dda2fe896d96afb294c6b69c2ebfab4b8"} Jul 24 03:31:43.692342 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:43.692284 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp" podStartSLOduration=1.530314972 podStartE2EDuration="1.692265164s" totalImagesPullingTime="161.950192ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:31:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:31:43.690614283 +0000 UTC m=+853.130033743" watchObservedRunningTime="2026-07-24 03:31:43.692265164 +0000 UTC m=+853.131684638" Jul 24 03:31:53.038631 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:53.038600 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp" Jul 24 03:31:53.039115 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:53.038646 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp" Jul 24 03:31:53.051433 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:53.051408 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp" Jul 24 03:31:53.735352 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:31:53.735324 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp" Jul 24 03:32:39.851218 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:32:39.851186 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:32:46.004634 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:32:46.004598 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:32:46.196407 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:32:46.196379 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:32:46.329539 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:32:46.329425 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:32:47.295829 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:32:47.295795 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:32:47.394228 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:32:47.394195 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:33:33.882700 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:33:33.882661 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-687994cdfd-tlnsg"] Jul 24 03:33:33.883242 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:33:33.882877 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-687994cdfd-tlnsg" podUID="ce3547aa-04d5-474a-a383-35be3fa725f8" containerName="manager" containerID="cri-o://2f0da35572a3bb566fcf2415b584bd9a26f8f30a4bbac1692ea0eff613ee250a" gracePeriod=30 Jul 24 03:33:34.138163 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:33:34.138100 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-687994cdfd-tlnsg" Jul 24 03:33:34.224499 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:33:34.224454 2579 generic.go:356] "Generic (PLEG): container finished" podID="ce3547aa-04d5-474a-a383-35be3fa725f8" containerID="2f0da35572a3bb566fcf2415b584bd9a26f8f30a4bbac1692ea0eff613ee250a" exitCode=0 Jul 24 03:33:34.224663 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:33:34.224520 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-687994cdfd-tlnsg" Jul 24 03:33:34.224663 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:33:34.224540 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-687994cdfd-tlnsg" event={"ID":"ce3547aa-04d5-474a-a383-35be3fa725f8","Type":"ContainerDied","Data":"2f0da35572a3bb566fcf2415b584bd9a26f8f30a4bbac1692ea0eff613ee250a"} Jul 24 03:33:34.224663 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:33:34.224583 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-687994cdfd-tlnsg" event={"ID":"ce3547aa-04d5-474a-a383-35be3fa725f8","Type":"ContainerDied","Data":"9b161ce0e22f0aa578217a79d827ff1124e691e41ceef6d06aacab65fec083d8"} Jul 24 03:33:34.224663 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:33:34.224606 2579 scope.go:117] "RemoveContainer" containerID="2f0da35572a3bb566fcf2415b584bd9a26f8f30a4bbac1692ea0eff613ee250a" Jul 24 03:33:34.233396 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:33:34.233376 2579 scope.go:117] "RemoveContainer" containerID="2f0da35572a3bb566fcf2415b584bd9a26f8f30a4bbac1692ea0eff613ee250a" Jul 24 03:33:34.233693 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:33:34.233674 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f0da35572a3bb566fcf2415b584bd9a26f8f30a4bbac1692ea0eff613ee250a\": container with ID starting with 2f0da35572a3bb566fcf2415b584bd9a26f8f30a4bbac1692ea0eff613ee250a not found: ID does not exist" containerID="2f0da35572a3bb566fcf2415b584bd9a26f8f30a4bbac1692ea0eff613ee250a" Jul 24 03:33:34.233753 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:33:34.233703 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0da35572a3bb566fcf2415b584bd9a26f8f30a4bbac1692ea0eff613ee250a"} err="failed to get container status \"2f0da35572a3bb566fcf2415b584bd9a26f8f30a4bbac1692ea0eff613ee250a\": rpc error: code = NotFound desc = could not find container \"2f0da35572a3bb566fcf2415b584bd9a26f8f30a4bbac1692ea0eff613ee250a\": container with ID starting with 2f0da35572a3bb566fcf2415b584bd9a26f8f30a4bbac1692ea0eff613ee250a not found: ID does not exist" Jul 24 03:33:34.247417 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:33:34.247392 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jhms\" (UniqueName: \"kubernetes.io/projected/ce3547aa-04d5-474a-a383-35be3fa725f8-kube-api-access-6jhms\") pod \"ce3547aa-04d5-474a-a383-35be3fa725f8\" (UID: \"ce3547aa-04d5-474a-a383-35be3fa725f8\") " Jul 24 03:33:34.247567 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:33:34.247517 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce3547aa-04d5-474a-a383-35be3fa725f8-webhook-cert\") pod \"ce3547aa-04d5-474a-a383-35be3fa725f8\" (UID: \"ce3547aa-04d5-474a-a383-35be3fa725f8\") " Jul 24 03:33:34.249631 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:33:34.249602 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce3547aa-04d5-474a-a383-35be3fa725f8-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ce3547aa-04d5-474a-a383-35be3fa725f8" (UID: "ce3547aa-04d5-474a-a383-35be3fa725f8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 03:33:34.249631 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:33:34.249613 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3547aa-04d5-474a-a383-35be3fa725f8-kube-api-access-6jhms" (OuterVolumeSpecName: "kube-api-access-6jhms") pod "ce3547aa-04d5-474a-a383-35be3fa725f8" (UID: "ce3547aa-04d5-474a-a383-35be3fa725f8"). InnerVolumeSpecName "kube-api-access-6jhms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:33:34.348503 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:33:34.348438 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jhms\" (UniqueName: \"kubernetes.io/projected/ce3547aa-04d5-474a-a383-35be3fa725f8-kube-api-access-6jhms\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:33:34.348503 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:33:34.348503 2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce3547aa-04d5-474a-a383-35be3fa725f8-webhook-cert\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:33:34.546822 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:33:34.546793 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-687994cdfd-tlnsg"] Jul 24 03:33:34.553042 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:33:34.553011 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-687994cdfd-tlnsg"] Jul 24 03:33:35.034190 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:33:35.034157 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce3547aa-04d5-474a-a383-35be3fa725f8" path="/var/lib/kubelet/pods/ce3547aa-04d5-474a-a383-35be3fa725f8/volumes" Jul 24 03:33:48.477937 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:33:48.477903 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:33:48.581515 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:33:48.581451 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:34:35.409967 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:35.409935 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-687994cdfd-2vhls"] Jul 24 03:34:35.410391 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:35.410327 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce3547aa-04d5-474a-a383-35be3fa725f8" containerName="manager" Jul 24 03:34:35.410391 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:35.410337 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce3547aa-04d5-474a-a383-35be3fa725f8" containerName="manager" Jul 24 03:34:35.410479 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:35.410424 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce3547aa-04d5-474a-a383-35be3fa725f8" containerName="manager" Jul 24 03:34:35.413566 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:35.413551 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-687994cdfd-2vhls" Jul 24 03:34:35.416972 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:35.416947 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-j56qj\"" Jul 24 03:34:35.417130 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:35.416988 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 03:34:35.417130 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:35.417015 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 03:34:35.426797 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:35.426773 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-687994cdfd-2vhls"] Jul 24 03:34:35.515538 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:35.515508 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96dr4\" (UniqueName: \"kubernetes.io/projected/45496562-6125-42ca-a0e1-ead0304a3aee-kube-api-access-96dr4\") pod \"maas-controller-687994cdfd-2vhls\" (UID: \"45496562-6125-42ca-a0e1-ead0304a3aee\") " pod="opendatahub/maas-controller-687994cdfd-2vhls" Jul 24 03:34:35.515738 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:35.515621 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45496562-6125-42ca-a0e1-ead0304a3aee-webhook-cert\") pod \"maas-controller-687994cdfd-2vhls\" (UID: \"45496562-6125-42ca-a0e1-ead0304a3aee\") " pod="opendatahub/maas-controller-687994cdfd-2vhls" Jul 24 03:34:35.616624 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:35.616585 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96dr4\" (UniqueName: \"kubernetes.io/projected/45496562-6125-42ca-a0e1-ead0304a3aee-kube-api-access-96dr4\") pod \"maas-controller-687994cdfd-2vhls\" (UID: \"45496562-6125-42ca-a0e1-ead0304a3aee\") " pod="opendatahub/maas-controller-687994cdfd-2vhls" Jul 24 03:34:35.616819 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:35.616650 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45496562-6125-42ca-a0e1-ead0304a3aee-webhook-cert\") pod \"maas-controller-687994cdfd-2vhls\" (UID: \"45496562-6125-42ca-a0e1-ead0304a3aee\") " pod="opendatahub/maas-controller-687994cdfd-2vhls" Jul 24 03:34:35.619312 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:35.619288 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45496562-6125-42ca-a0e1-ead0304a3aee-webhook-cert\") pod \"maas-controller-687994cdfd-2vhls\" (UID: \"45496562-6125-42ca-a0e1-ead0304a3aee\") " pod="opendatahub/maas-controller-687994cdfd-2vhls" Jul 24 03:34:35.624592 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:35.624570 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96dr4\" (UniqueName: \"kubernetes.io/projected/45496562-6125-42ca-a0e1-ead0304a3aee-kube-api-access-96dr4\") pod \"maas-controller-687994cdfd-2vhls\" (UID: \"45496562-6125-42ca-a0e1-ead0304a3aee\") " pod="opendatahub/maas-controller-687994cdfd-2vhls" Jul 24 03:34:35.723759 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:35.723680 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-687994cdfd-2vhls" Jul 24 03:34:35.857043 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:35.856987 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-687994cdfd-2vhls"] Jul 24 03:34:35.859329 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:34:35.859303 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45496562_6125_42ca_a0e1_ead0304a3aee.slice/crio-ae79062e75eca7d0e03abd619e765bac39b8b888e131fd29ee2cfe9ae3e772ff WatchSource:0}: Error finding container ae79062e75eca7d0e03abd619e765bac39b8b888e131fd29ee2cfe9ae3e772ff: Status 404 returned error can't find the container with id ae79062e75eca7d0e03abd619e765bac39b8b888e131fd29ee2cfe9ae3e772ff Jul 24 03:34:35.860890 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:35.860873 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 03:34:36.524306 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:36.524207 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-687994cdfd-2vhls" event={"ID":"45496562-6125-42ca-a0e1-ead0304a3aee","Type":"ContainerStarted","Data":"7f12b037085569ea4a5780d851a9741f746d120fe8bd48ffd02bf67175681718"} Jul 24 03:34:36.524306 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:36.524276 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-687994cdfd-2vhls" event={"ID":"45496562-6125-42ca-a0e1-ead0304a3aee","Type":"ContainerStarted","Data":"ae79062e75eca7d0e03abd619e765bac39b8b888e131fd29ee2cfe9ae3e772ff"} Jul 24 03:34:36.524306 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:36.524303 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-687994cdfd-2vhls" Jul 24 03:34:36.541991 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:36.541948 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-687994cdfd-2vhls" podStartSLOduration=1.145690432 podStartE2EDuration="1.541931993s" totalImagesPullingTime="396.241561ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:34:36.539329485 +0000 UTC m=+1025.978748944" watchObservedRunningTime="2026-07-24 03:34:36.541931993 +0000 UTC m=+1025.981351454" Jul 24 03:34:47.533814 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:47.533781 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-687994cdfd-2vhls" Jul 24 03:34:52.081345 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:52.081307 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:34:52.197558 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:52.197524 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-687994cdfd-2vhls"] Jul 24 03:34:52.197752 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:52.197716 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-687994cdfd-2vhls" podUID="45496562-6125-42ca-a0e1-ead0304a3aee" containerName="manager" containerID="cri-o://7f12b037085569ea4a5780d851a9741f746d120fe8bd48ffd02bf67175681718" gracePeriod=30 Jul 24 03:34:52.449808 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:52.449782 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-687994cdfd-2vhls" Jul 24 03:34:52.578414 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:52.578387 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96dr4\" (UniqueName: \"kubernetes.io/projected/45496562-6125-42ca-a0e1-ead0304a3aee-kube-api-access-96dr4\") pod \"45496562-6125-42ca-a0e1-ead0304a3aee\" (UID: \"45496562-6125-42ca-a0e1-ead0304a3aee\") " Jul 24 03:34:52.578599 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:52.578437 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45496562-6125-42ca-a0e1-ead0304a3aee-webhook-cert\") pod \"45496562-6125-42ca-a0e1-ead0304a3aee\" (UID: \"45496562-6125-42ca-a0e1-ead0304a3aee\") " Jul 24 03:34:52.580613 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:52.580582 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45496562-6125-42ca-a0e1-ead0304a3aee-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "45496562-6125-42ca-a0e1-ead0304a3aee" (UID: "45496562-6125-42ca-a0e1-ead0304a3aee"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 03:34:52.580749 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:52.580688 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45496562-6125-42ca-a0e1-ead0304a3aee-kube-api-access-96dr4" (OuterVolumeSpecName: "kube-api-access-96dr4") pod "45496562-6125-42ca-a0e1-ead0304a3aee" (UID: "45496562-6125-42ca-a0e1-ead0304a3aee"). InnerVolumeSpecName "kube-api-access-96dr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:34:52.610153 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:52.610122 2579 generic.go:356] "Generic (PLEG): container finished" podID="45496562-6125-42ca-a0e1-ead0304a3aee" containerID="7f12b037085569ea4a5780d851a9741f746d120fe8bd48ffd02bf67175681718" exitCode=0 Jul 24 03:34:52.610262 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:52.610165 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-687994cdfd-2vhls" Jul 24 03:34:52.610262 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:52.610205 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-687994cdfd-2vhls" event={"ID":"45496562-6125-42ca-a0e1-ead0304a3aee","Type":"ContainerDied","Data":"7f12b037085569ea4a5780d851a9741f746d120fe8bd48ffd02bf67175681718"} Jul 24 03:34:52.610262 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:52.610257 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-687994cdfd-2vhls" event={"ID":"45496562-6125-42ca-a0e1-ead0304a3aee","Type":"ContainerDied","Data":"ae79062e75eca7d0e03abd619e765bac39b8b888e131fd29ee2cfe9ae3e772ff"} Jul 24 03:34:52.610378 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:52.610279 2579 scope.go:117] "RemoveContainer" containerID="7f12b037085569ea4a5780d851a9741f746d120fe8bd48ffd02bf67175681718" Jul 24 03:34:52.619977 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:52.619958 2579 scope.go:117] "RemoveContainer" containerID="7f12b037085569ea4a5780d851a9741f746d120fe8bd48ffd02bf67175681718" Jul 24 03:34:52.620231 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:34:52.620212 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f12b037085569ea4a5780d851a9741f746d120fe8bd48ffd02bf67175681718\": container with ID starting with 7f12b037085569ea4a5780d851a9741f746d120fe8bd48ffd02bf67175681718 not found: ID does not exist" containerID="7f12b037085569ea4a5780d851a9741f746d120fe8bd48ffd02bf67175681718" Jul 24 03:34:52.620288 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:52.620239 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f12b037085569ea4a5780d851a9741f746d120fe8bd48ffd02bf67175681718"} err="failed to get container status \"7f12b037085569ea4a5780d851a9741f746d120fe8bd48ffd02bf67175681718\": rpc error: code = NotFound desc = could not find container \"7f12b037085569ea4a5780d851a9741f746d120fe8bd48ffd02bf67175681718\": container with ID starting with 7f12b037085569ea4a5780d851a9741f746d120fe8bd48ffd02bf67175681718 not found: ID does not exist" Jul 24 03:34:52.632617 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:52.632592 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-687994cdfd-2vhls"] Jul 24 03:34:52.635621 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:52.635604 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-687994cdfd-2vhls"] Jul 24 03:34:52.679633 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:52.679605 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96dr4\" (UniqueName: \"kubernetes.io/projected/45496562-6125-42ca-a0e1-ead0304a3aee-kube-api-access-96dr4\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:34:52.679633 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:52.679631 2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45496562-6125-42ca-a0e1-ead0304a3aee-webhook-cert\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:34:53.034356 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:53.034272 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45496562-6125-42ca-a0e1-ead0304a3aee" path="/var/lib/kubelet/pods/45496562-6125-42ca-a0e1-ead0304a3aee/volumes" Jul 24 03:34:53.346838 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:53.346756 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:34:53.465035 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:53.465004 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-687994cdfd-m8bvr"] Jul 24 03:34:53.465409 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:53.465396 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45496562-6125-42ca-a0e1-ead0304a3aee" containerName="manager" Jul 24 03:34:53.465453 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:53.465412 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="45496562-6125-42ca-a0e1-ead0304a3aee" containerName="manager" Jul 24 03:34:53.465517 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:53.465504 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="45496562-6125-42ca-a0e1-ead0304a3aee" containerName="manager" Jul 24 03:34:53.469915 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:53.469894 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-687994cdfd-m8bvr" Jul 24 03:34:53.472723 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:53.472705 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 03:34:53.473662 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:53.473644 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 03:34:53.473718 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:53.473661 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-j56qj\"" Jul 24 03:34:53.479764 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:53.479729 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-687994cdfd-m8bvr"] Jul 24 03:34:53.589287 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:53.589248 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4998e24-36bc-49d8-968d-0f60ae219e0b-webhook-cert\") pod \"maas-controller-687994cdfd-m8bvr\" (UID: \"c4998e24-36bc-49d8-968d-0f60ae219e0b\") " pod="opendatahub/maas-controller-687994cdfd-m8bvr" Jul 24 03:34:53.589527 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:53.589299 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2zwj\" (UniqueName: \"kubernetes.io/projected/c4998e24-36bc-49d8-968d-0f60ae219e0b-kube-api-access-s2zwj\") pod \"maas-controller-687994cdfd-m8bvr\" (UID: \"c4998e24-36bc-49d8-968d-0f60ae219e0b\") " pod="opendatahub/maas-controller-687994cdfd-m8bvr" Jul 24 03:34:53.690696 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:53.690643 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4998e24-36bc-49d8-968d-0f60ae219e0b-webhook-cert\") pod \"maas-controller-687994cdfd-m8bvr\" (UID: \"c4998e24-36bc-49d8-968d-0f60ae219e0b\") " pod="opendatahub/maas-controller-687994cdfd-m8bvr" Jul 24 03:34:53.690887 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:53.690712 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2zwj\" (UniqueName: \"kubernetes.io/projected/c4998e24-36bc-49d8-968d-0f60ae219e0b-kube-api-access-s2zwj\") pod \"maas-controller-687994cdfd-m8bvr\" (UID: \"c4998e24-36bc-49d8-968d-0f60ae219e0b\") " pod="opendatahub/maas-controller-687994cdfd-m8bvr" Jul 24 03:34:53.693273 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:53.693243 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4998e24-36bc-49d8-968d-0f60ae219e0b-webhook-cert\") pod \"maas-controller-687994cdfd-m8bvr\" (UID: \"c4998e24-36bc-49d8-968d-0f60ae219e0b\") " pod="opendatahub/maas-controller-687994cdfd-m8bvr" Jul 24 03:34:53.698888 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:53.698861 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2zwj\" (UniqueName: \"kubernetes.io/projected/c4998e24-36bc-49d8-968d-0f60ae219e0b-kube-api-access-s2zwj\") pod \"maas-controller-687994cdfd-m8bvr\" (UID: \"c4998e24-36bc-49d8-968d-0f60ae219e0b\") " pod="opendatahub/maas-controller-687994cdfd-m8bvr" Jul 24 03:34:53.781735 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:53.781703 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-687994cdfd-m8bvr" Jul 24 03:34:53.916258 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:53.916234 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-687994cdfd-m8bvr"] Jul 24 03:34:53.918315 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:34:53.918291 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4998e24_36bc_49d8_968d_0f60ae219e0b.slice/crio-ee4e0db0d784d7ec4f7f3358fe0cbe79f758f0796b7f3c860027ea1fa090d025 WatchSource:0}: Error finding container ee4e0db0d784d7ec4f7f3358fe0cbe79f758f0796b7f3c860027ea1fa090d025: Status 404 returned error can't find the container with id ee4e0db0d784d7ec4f7f3358fe0cbe79f758f0796b7f3c860027ea1fa090d025 Jul 24 03:34:54.621999 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:54.621959 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-687994cdfd-m8bvr" event={"ID":"c4998e24-36bc-49d8-968d-0f60ae219e0b","Type":"ContainerStarted","Data":"093cc38f24716f48c5e933b30260f61e2c868cc5d092a67be468fc3add25b7e7"} Jul 24 03:34:54.622486 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:54.622016 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-687994cdfd-m8bvr" event={"ID":"c4998e24-36bc-49d8-968d-0f60ae219e0b","Type":"ContainerStarted","Data":"ee4e0db0d784d7ec4f7f3358fe0cbe79f758f0796b7f3c860027ea1fa090d025"} Jul 24 03:34:54.622486 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:54.622048 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-687994cdfd-m8bvr" Jul 24 03:34:54.640018 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:54.639963 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-687994cdfd-m8bvr" podStartSLOduration=1.2527651450000001 podStartE2EDuration="1.639948315s" totalImagesPullingTime="387.18317ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:34:54.637819232 +0000 UTC m=+1044.077238691" watchObservedRunningTime="2026-07-24 03:34:54.639948315 +0000 UTC m=+1044.079367775" Jul 24 03:34:58.539018 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:58.538979 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:34:58.668877 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:34:58.668834 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:35:05.629831 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:35:05.629803 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-687994cdfd-m8bvr" Jul 24 03:35:36.232751 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:35:36.232702 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:35:36.335993 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:35:36.335957 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:35:36.443821 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:35:36.443791 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:35:42.424916 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:35:42.424865 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:35:42.492588 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:35:42.492526 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:35:42.571907 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:35:42.571876 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:35:42.672044 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:35:42.672018 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:35:42.799279 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:35:42.799205 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:35:48.573621 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:35:48.573588 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:35:48.677353 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:35:48.677308 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:36:45.682055 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:36:45.682014 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:36:45.751052 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:36:45.751018 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:38:03.595638 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:38:03.595596 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:38:04.756599 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:38:04.756567 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:38:14.333325 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:38:14.333294 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:38:14.435141 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:38:14.435108 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:39:04.878353 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:39:04.878318 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:42:02.283636 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:42:02.283600 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:42:02.370617 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:42:02.370584 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:43:31.377549 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:43:31.377512 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:43:41.941755 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:43:41.941718 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:44:07.391971 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:44:07.391936 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:44:08.449584 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:44:08.449541 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:44:12.616482 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:44:12.616419 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:44:12.717198 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:44:12.717164 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:45:00.145484 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:45:00.145426 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747745-xzbmt"] Jul 24 03:45:00.148985 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:45:00.148968 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747745-xzbmt" Jul 24 03:45:00.151476 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:45:00.151441 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgb7g\"" Jul 24 03:45:00.155445 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:45:00.155420 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747745-xzbmt"] Jul 24 03:45:00.159847 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:45:00.159827 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747745-xzbmt" Jul 24 03:45:00.510871 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:45:00.510848 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747745-xzbmt"] Jul 24 03:45:00.517713 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:45:00.517692 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 03:45:00.671272 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:45:00.671232 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747745-xzbmt" event={"ID":"1610a97e-cf52-4955-a551-8de63aedf990","Type":"ContainerStarted","Data":"a4592e9992930878296711a729aad22273d7e9181bee6f7e30d1d57dcf4215f2"} Jul 24 03:45:03.688639 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:45:03.688603 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747745-xzbmt" event={"ID":"1610a97e-cf52-4955-a551-8de63aedf990","Type":"ContainerStarted","Data":"23baeeb1026cc32e943bc7ddf6eea81ada9258bcff1cf681e44c6d716ae89fd0"} Jul 24 03:45:03.705510 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:45:03.705448 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747745-xzbmt" podStartSLOduration=1.027825581 podStartE2EDuration="3.705433634s" totalImagesPullingTime="2.677608053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:45:03.703975686 +0000 UTC m=+1653.143395146" watchObservedRunningTime="2026-07-24 03:45:03.705433634 +0000 UTC m=+1653.144853095" Jul 24 03:45:04.695408 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:45:04.695365 2579 generic.go:356] "Generic (PLEG): container finished" podID="1610a97e-cf52-4955-a551-8de63aedf990" containerID="23baeeb1026cc32e943bc7ddf6eea81ada9258bcff1cf681e44c6d716ae89fd0" exitCode=0 Jul 24 03:45:04.695820 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:45:04.695456 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747745-xzbmt" event={"ID":"1610a97e-cf52-4955-a551-8de63aedf990","Type":"ContainerDied","Data":"23baeeb1026cc32e943bc7ddf6eea81ada9258bcff1cf681e44c6d716ae89fd0"} Jul 24 03:45:05.852404 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:45:05.852380 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747745-xzbmt" Jul 24 03:45:06.706866 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:45:06.706834 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747745-xzbmt" Jul 24 03:45:06.707043 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:45:06.706888 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747745-xzbmt" event={"ID":"1610a97e-cf52-4955-a551-8de63aedf990","Type":"ContainerDied","Data":"a4592e9992930878296711a729aad22273d7e9181bee6f7e30d1d57dcf4215f2"} Jul 24 03:45:06.707043 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:45:06.706946 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4592e9992930878296711a729aad22273d7e9181bee6f7e30d1d57dcf4215f2" Jul 24 03:47:03.982917 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:03.982884 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:47:16.751063 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:16.751024 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:47:36.173438 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:36.173402 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:47:39.400237 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:39.400156 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:47:39.487265 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:39.487232 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:47:39.887444 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:39.887411 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:47:45.541736 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:45.541698 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:47:45.642119 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:45.642082 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:47:45.789602 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:45.789561 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:47:46.089509 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.089474 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp"] Jul 24 03:47:46.090098 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.090083 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1610a97e-cf52-4955-a551-8de63aedf990" containerName="cleanup" Jul 24 03:47:46.090147 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.090101 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1610a97e-cf52-4955-a551-8de63aedf990" containerName="cleanup" Jul 24 03:47:46.090204 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.090193 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1610a97e-cf52-4955-a551-8de63aedf990" containerName="cleanup" Jul 24 03:47:46.093170 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.093150 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.095588 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.095565 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3b0f86fb-openshift-default-dockercfg-qppsb\"" Jul 24 03:47:46.108240 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.108210 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:47:46.110743 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.110721 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp"] Jul 24 03:47:46.156629 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.156595 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-workload-certs\") pod \"e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.156629 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.156633 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-istio-envoy\") pod \"e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.156833 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.156703 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e54f2362-710a-443b-b5b1-c47e529bfac1-istio-podinfo\") pod \"e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.156833 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.156724 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf949\" (UniqueName: \"kubernetes.io/projected/e54f2362-710a-443b-b5b1-c47e529bfac1-kube-api-access-tf949\") pod \"e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.156833 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.156748 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-workload-socket\") pod \"e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.156833 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.156785 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e54f2362-710a-443b-b5b1-c47e529bfac1-istio-token\") pod \"e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.156833 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.156819 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-istio-data\") pod \"e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.157079 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.156880 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e54f2362-710a-443b-b5b1-c47e529bfac1-istiod-ca-cert\") pod \"e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.157079 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.157047 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-credential-socket\") pod \"e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.257930 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.257895 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e54f2362-710a-443b-b5b1-c47e529bfac1-istiod-ca-cert\") pod \"e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.258097 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.257945 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-credential-socket\") pod \"e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.258097 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.257983 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-workload-certs\") pod \"e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.258097 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.257999 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-istio-envoy\") pod \"e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.258097 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.258028 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e54f2362-710a-443b-b5b1-c47e529bfac1-istio-podinfo\") pod \"e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.258097 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.258055 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf949\" (UniqueName: \"kubernetes.io/projected/e54f2362-710a-443b-b5b1-c47e529bfac1-kube-api-access-tf949\") pod \"e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.258371 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.258216 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-workload-socket\") pod \"e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.258371 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.258258 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e54f2362-710a-443b-b5b1-c47e529bfac1-istio-token\") pod \"e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.258371 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.258293 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-istio-data\") pod \"e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.258620 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.258594 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-credential-socket\") pod \"e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.258855 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.258736 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e54f2362-710a-443b-b5b1-c47e529bfac1-istiod-ca-cert\") pod \"e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.258984 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.258867 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-workload-socket\") pod \"e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.258984 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.258827 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-istio-data\") pod \"e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.259288 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.259268 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-workload-certs\") pod \"e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.260537 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.260517 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-istio-envoy\") pod \"e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.261120 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.261095 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e54f2362-710a-443b-b5b1-c47e529bfac1-istio-podinfo\") pod \"e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.266247 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.266220 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf949\" (UniqueName: \"kubernetes.io/projected/e54f2362-710a-443b-b5b1-c47e529bfac1-kube-api-access-tf949\") pod \"e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.266328 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.266300 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e54f2362-710a-443b-b5b1-c47e529bfac1-istio-token\") pod \"e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.359924 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.359892 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:47:46.403881 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.403844 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:46.547405 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.547379 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp"] Jul 24 03:47:46.549623 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:47:46.549593 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode54f2362_710a_443b_b5b1_c47e529bfac1.slice/crio-b43385906b64e095739de091208a638c556e314ecb1355a9f6f55aa32dd5c1c2 WatchSource:0}: Error finding container b43385906b64e095739de091208a638c556e314ecb1355a9f6f55aa32dd5c1c2: Status 404 returned error can't find the container with id b43385906b64e095739de091208a638c556e314ecb1355a9f6f55aa32dd5c1c2 Jul 24 03:47:46.551877 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.551845 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:47:46.551964 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.551911 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:47:46.551964 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:46.551944 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:47:47.509031 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:47.508997 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" event={"ID":"e54f2362-710a-443b-b5b1-c47e529bfac1","Type":"ContainerStarted","Data":"3c989dd846d6e79c5d50c29516f3f327f79256d1fef811d0ff4f1abe45a5ccc5"} Jul 24 03:47:47.509232 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:47.509048 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" event={"ID":"e54f2362-710a-443b-b5b1-c47e529bfac1","Type":"ContainerStarted","Data":"b43385906b64e095739de091208a638c556e314ecb1355a9f6f55aa32dd5c1c2"} Jul 24 03:47:47.536506 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:47.536436 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" podStartSLOduration=1.536420876 podStartE2EDuration="1.536420876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:47:47.535150208 +0000 UTC m=+1816.974569681" watchObservedRunningTime="2026-07-24 03:47:47.536420876 +0000 UTC m=+1816.975840336" Jul 24 03:47:48.404630 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:48.404591 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:48.409323 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:48.409301 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:48.514705 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:48.514672 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:48.515605 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:48.515583 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:47:48.869012 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:48.868976 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:47:48.970751 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:48.970708 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:47:49.054933 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:49.054901 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz"] Jul 24 03:47:49.058816 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:49.058794 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz" Jul 24 03:47:49.061130 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:49.061110 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3b0f86fb-serving-cert\"" Jul 24 03:47:49.072003 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:49.071834 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz"] Jul 24 03:47:49.106336 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:49.106303 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:47:49.189314 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:49.189233 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmfd9\" (UniqueName: \"kubernetes.io/projected/168053f1-96db-49a5-94e1-d0d521761baa-kube-api-access-vmfd9\") pod \"maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz\" (UID: \"168053f1-96db-49a5-94e1-d0d521761baa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz" Jul 24 03:47:49.189314 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:49.189268 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/168053f1-96db-49a5-94e1-d0d521761baa-maas-api-tls\") pod \"maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz\" (UID: \"168053f1-96db-49a5-94e1-d0d521761baa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz" Jul 24 03:47:49.189547 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:49.189339 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/168053f1-96db-49a5-94e1-d0d521761baa-openshift-service-ca\") pod \"maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz\" (UID: \"168053f1-96db-49a5-94e1-d0d521761baa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz" Jul 24 03:47:49.205402 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:49.205356 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:47:49.290028 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:49.289993 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmfd9\" (UniqueName: \"kubernetes.io/projected/168053f1-96db-49a5-94e1-d0d521761baa-kube-api-access-vmfd9\") pod \"maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz\" (UID: \"168053f1-96db-49a5-94e1-d0d521761baa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz" Jul 24 03:47:49.290028 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:49.290026 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/168053f1-96db-49a5-94e1-d0d521761baa-maas-api-tls\") pod \"maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz\" (UID: \"168053f1-96db-49a5-94e1-d0d521761baa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz" Jul 24 03:47:49.290227 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:49.290073 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/168053f1-96db-49a5-94e1-d0d521761baa-openshift-service-ca\") pod \"maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz\" (UID: \"168053f1-96db-49a5-94e1-d0d521761baa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz" Jul 24 03:47:49.290883 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:49.290855 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/168053f1-96db-49a5-94e1-d0d521761baa-openshift-service-ca\") pod \"maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz\" (UID: \"168053f1-96db-49a5-94e1-d0d521761baa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz" Jul 24 03:47:49.292695 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:49.292671 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/168053f1-96db-49a5-94e1-d0d521761baa-maas-api-tls\") pod \"maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz\" (UID: \"168053f1-96db-49a5-94e1-d0d521761baa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz" Jul 24 03:47:49.298103 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:49.298080 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmfd9\" (UniqueName: \"kubernetes.io/projected/168053f1-96db-49a5-94e1-d0d521761baa-kube-api-access-vmfd9\") pod \"maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz\" (UID: \"168053f1-96db-49a5-94e1-d0d521761baa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz" Jul 24 03:47:49.373916 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:49.373886 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz" Jul 24 03:47:49.518005 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:49.517977 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz"] Jul 24 03:47:49.519104 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:47:49.519077 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod168053f1_96db_49a5_94e1_d0d521761baa.slice/crio-be42279090cb8e9b20b0fa18766de21bdd13fb48599d7870c5c1c342fceaf798 WatchSource:0}: Error finding container be42279090cb8e9b20b0fa18766de21bdd13fb48599d7870c5c1c342fceaf798: Status 404 returned error can't find the container with id be42279090cb8e9b20b0fa18766de21bdd13fb48599d7870c5c1c342fceaf798 Jul 24 03:47:49.590453 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:49.590422 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:47:49.696660 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:49.696631 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:47:50.526683 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:50.526643 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz" event={"ID":"168053f1-96db-49a5-94e1-d0d521761baa","Type":"ContainerStarted","Data":"cf049dafb2a8fa2d098dffa34a91d44bf800c6a00caa629f4954811bec577f9e"} Jul 24 03:47:50.527165 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:50.526704 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz" event={"ID":"168053f1-96db-49a5-94e1-d0d521761baa","Type":"ContainerStarted","Data":"be42279090cb8e9b20b0fa18766de21bdd13fb48599d7870c5c1c342fceaf798"} Jul 24 03:47:50.527165 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:50.526904 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz" Jul 24 03:47:50.551348 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:50.551297 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz" podStartSLOduration=1.078716937 podStartE2EDuration="1.551282153s" totalImagesPullingTime="472.565216ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:47:50.547726619 +0000 UTC m=+1819.987146079" watchObservedRunningTime="2026-07-24 03:47:50.551282153 +0000 UTC m=+1819.990701649" Jul 24 03:47:56.537420 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:47:56.537389 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz" Jul 24 03:48:04.556649 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:04.556609 2579 patch_prober.go:28] interesting pod/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 03:48:04.557056 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:04.556687 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" podUID="e54f2362-710a-443b-b5b1-c47e529bfac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 03:48:05.612710 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:05.612678 2579 generic.go:356] "Generic (PLEG): container finished" podID="e54f2362-710a-443b-b5b1-c47e529bfac1" containerID="3c989dd846d6e79c5d50c29516f3f327f79256d1fef811d0ff4f1abe45a5ccc5" exitCode=0 Jul 24 03:48:05.613167 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:05.612758 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" event={"ID":"e54f2362-710a-443b-b5b1-c47e529bfac1","Type":"ContainerDied","Data":"3c989dd846d6e79c5d50c29516f3f327f79256d1fef811d0ff4f1abe45a5ccc5"} Jul 24 03:48:05.613211 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:05.613179 2579 scope.go:117] "RemoveContainer" containerID="3c989dd846d6e79c5d50c29516f3f327f79256d1fef811d0ff4f1abe45a5ccc5" Jul 24 03:48:05.614238 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:05.614208 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:48:05.614329 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:05.614276 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:48:05.614329 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:05.614307 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:48:06.619357 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:06.619323 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" event={"ID":"e54f2362-710a-443b-b5b1-c47e529bfac1","Type":"ContainerStarted","Data":"051e83cf6fd8fc0711fe2f66aff2228d10d5b3aa530402d6ccbcfa0ccd90f973"} Jul 24 03:48:07.404675 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:07.404640 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:48:07.404886 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:07.404866 2579 patch_prober.go:28] interesting pod/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 24 03:48:07.404935 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:07.404920 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" podUID="e54f2362-710a-443b-b5b1-c47e529bfac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 24 03:48:08.405106 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:08.405060 2579 patch_prober.go:28] interesting pod/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 24 03:48:08.405519 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:08.405122 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" podUID="e54f2362-710a-443b-b5b1-c47e529bfac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 24 03:48:09.404572 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:09.404531 2579 patch_prober.go:28] interesting pod/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 24 03:48:09.404750 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:09.404602 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" podUID="e54f2362-710a-443b-b5b1-c47e529bfac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 24 03:48:10.405117 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:10.405071 2579 patch_prober.go:28] interesting pod/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 24 03:48:10.405568 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:10.405143 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" podUID="e54f2362-710a-443b-b5b1-c47e529bfac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 24 03:48:11.407765 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:11.407730 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:48:11.408232 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:11.407989 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:48:11.408836 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:11.408814 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:48:24.923770 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:24.923733 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz"] Jul 24 03:48:24.924234 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:24.923992 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz" podUID="168053f1-96db-49a5-94e1-d0d521761baa" containerName="maas-api" containerID="cri-o://cf049dafb2a8fa2d098dffa34a91d44bf800c6a00caa629f4954811bec577f9e" gracePeriod=30 Jul 24 03:48:24.950835 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:24.950806 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:48:25.022312 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:25.022269 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:48:25.094425 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:25.094379 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:48:25.185912 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:25.185848 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz" Jul 24 03:48:25.249238 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:25.249208 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/168053f1-96db-49a5-94e1-d0d521761baa-maas-api-tls\") pod \"168053f1-96db-49a5-94e1-d0d521761baa\" (UID: \"168053f1-96db-49a5-94e1-d0d521761baa\") " Jul 24 03:48:25.249238 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:25.249240 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/168053f1-96db-49a5-94e1-d0d521761baa-openshift-service-ca\") pod \"168053f1-96db-49a5-94e1-d0d521761baa\" (UID: \"168053f1-96db-49a5-94e1-d0d521761baa\") " Jul 24 03:48:25.249506 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:25.249261 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmfd9\" (UniqueName: \"kubernetes.io/projected/168053f1-96db-49a5-94e1-d0d521761baa-kube-api-access-vmfd9\") pod \"168053f1-96db-49a5-94e1-d0d521761baa\" (UID: \"168053f1-96db-49a5-94e1-d0d521761baa\") " Jul 24 03:48:25.249616 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:25.249593 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/168053f1-96db-49a5-94e1-d0d521761baa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "168053f1-96db-49a5-94e1-d0d521761baa" (UID: "168053f1-96db-49a5-94e1-d0d521761baa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 03:48:25.251666 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:25.251642 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/168053f1-96db-49a5-94e1-d0d521761baa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "168053f1-96db-49a5-94e1-d0d521761baa" (UID: "168053f1-96db-49a5-94e1-d0d521761baa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 03:48:25.251782 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:25.251665 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168053f1-96db-49a5-94e1-d0d521761baa-kube-api-access-vmfd9" (OuterVolumeSpecName: "kube-api-access-vmfd9") pod "168053f1-96db-49a5-94e1-d0d521761baa" (UID: "168053f1-96db-49a5-94e1-d0d521761baa"). InnerVolumeSpecName "kube-api-access-vmfd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:48:25.350659 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:25.350626 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/168053f1-96db-49a5-94e1-d0d521761baa-maas-api-tls\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:48:25.350659 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:25.350655 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/168053f1-96db-49a5-94e1-d0d521761baa-openshift-service-ca\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:48:25.350830 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:25.350671 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmfd9\" (UniqueName: \"kubernetes.io/projected/168053f1-96db-49a5-94e1-d0d521761baa-kube-api-access-vmfd9\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:48:25.715806 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:25.715771 2579 generic.go:356] "Generic (PLEG): container finished" podID="168053f1-96db-49a5-94e1-d0d521761baa" containerID="cf049dafb2a8fa2d098dffa34a91d44bf800c6a00caa629f4954811bec577f9e" exitCode=0 Jul 24 03:48:25.715975 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:25.715818 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz" Jul 24 03:48:25.715975 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:25.715857 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz" event={"ID":"168053f1-96db-49a5-94e1-d0d521761baa","Type":"ContainerDied","Data":"cf049dafb2a8fa2d098dffa34a91d44bf800c6a00caa629f4954811bec577f9e"} Jul 24 03:48:25.715975 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:25.715919 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz" event={"ID":"168053f1-96db-49a5-94e1-d0d521761baa","Type":"ContainerDied","Data":"be42279090cb8e9b20b0fa18766de21bdd13fb48599d7870c5c1c342fceaf798"} Jul 24 03:48:25.715975 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:25.715943 2579 scope.go:117] "RemoveContainer" containerID="cf049dafb2a8fa2d098dffa34a91d44bf800c6a00caa629f4954811bec577f9e" Jul 24 03:48:25.725353 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:25.725337 2579 scope.go:117] "RemoveContainer" containerID="cf049dafb2a8fa2d098dffa34a91d44bf800c6a00caa629f4954811bec577f9e" Jul 24 03:48:25.725616 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:48:25.725599 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf049dafb2a8fa2d098dffa34a91d44bf800c6a00caa629f4954811bec577f9e\": container with ID starting with cf049dafb2a8fa2d098dffa34a91d44bf800c6a00caa629f4954811bec577f9e not found: ID does not exist" containerID="cf049dafb2a8fa2d098dffa34a91d44bf800c6a00caa629f4954811bec577f9e" Jul 24 03:48:25.725681 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:25.725623 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf049dafb2a8fa2d098dffa34a91d44bf800c6a00caa629f4954811bec577f9e"} err="failed to get container status \"cf049dafb2a8fa2d098dffa34a91d44bf800c6a00caa629f4954811bec577f9e\": rpc error: code = NotFound desc = could not find container \"cf049dafb2a8fa2d098dffa34a91d44bf800c6a00caa629f4954811bec577f9e\": container with ID starting with cf049dafb2a8fa2d098dffa34a91d44bf800c6a00caa629f4954811bec577f9e not found: ID does not exist" Jul 24 03:48:25.736664 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:25.736643 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz"] Jul 24 03:48:25.741081 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:25.741062 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3b0f86fb-5c6687bd86-fqxrz"] Jul 24 03:48:27.033521 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:27.033454 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="168053f1-96db-49a5-94e1-d0d521761baa" path="/var/lib/kubelet/pods/168053f1-96db-49a5-94e1-d0d521761baa/volumes" Jul 24 03:48:29.616692 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:29.616658 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:48:29.752104 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:29.752075 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp"] Jul 24 03:48:29.752344 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:29.752299 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" podUID="e54f2362-710a-443b-b5b1-c47e529bfac1" containerName="istio-proxy" containerID="cri-o://051e83cf6fd8fc0711fe2f66aff2228d10d5b3aa530402d6ccbcfa0ccd90f973" gracePeriod=30 Jul 24 03:48:29.773315 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:29.773285 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:48:30.031041 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:30.030952 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:48:30.279656 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:30.279619 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:48:30.390375 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:30.390345 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:48:30.542370 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:30.542342 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:48:30.812982 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:30.812899 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp"] Jul 24 03:48:30.813524 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:30.813506 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="168053f1-96db-49a5-94e1-d0d521761baa" containerName="maas-api" Jul 24 03:48:30.813624 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:30.813526 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="168053f1-96db-49a5-94e1-d0d521761baa" containerName="maas-api" Jul 24 03:48:30.813680 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:30.813631 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="168053f1-96db-49a5-94e1-d0d521761baa" containerName="maas-api" Jul 24 03:48:30.818021 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:30.818000 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:30.822338 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:30.822319 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-781d8b17-gw-openshift-default-dockercfg-rbgnp\"" Jul 24 03:48:30.831672 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:30.831648 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp"] Jul 24 03:48:30.846255 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:30.846226 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:48:30.901913 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:30.901880 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-workload-socket\") pod \"e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:30.902093 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:30.901948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istio-podinfo\") pod \"e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:30.902093 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:30.901981 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-credential-socket\") pod \"e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:30.902093 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:30.902011 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istio-envoy\") pod \"e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:30.902093 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:30.902042 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istiod-ca-cert\") pod \"e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:30.902298 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:30.902121 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-workload-certs\") pod \"e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:30.902298 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:30.902174 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istio-token\") pod \"e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:30.902298 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:30.902245 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istio-data\") pod \"e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:30.902298 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:30.902278 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdr5v\" (UniqueName: \"kubernetes.io/projected/1e4d01cd-4ac8-46af-bcc9-57708025ff82-kube-api-access-jdr5v\") pod \"e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:30.960084 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:30.960036 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:48:31.003182 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:31.003147 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istio-podinfo\") pod \"e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:31.003376 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:31.003197 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-credential-socket\") pod \"e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:31.003376 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:31.003368 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istio-envoy\") pod \"e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:31.003531 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:31.003415 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istiod-ca-cert\") pod \"e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:31.003531 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:31.003472 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-workload-certs\") pod \"e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:31.003632 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:31.003531 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istio-token\") pod \"e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:31.003686 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:31.003628 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istio-data\") pod \"e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:31.003686 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:31.003670 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdr5v\" (UniqueName: \"kubernetes.io/projected/1e4d01cd-4ac8-46af-bcc9-57708025ff82-kube-api-access-jdr5v\") pod \"e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:31.003686 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:31.003675 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-credential-socket\") pod \"e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:31.003844 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:31.003725 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-workload-socket\") pod \"e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:31.003899 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:31.003840 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-workload-certs\") pod \"e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:31.004070 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:31.004051 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-workload-socket\") pod \"e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:31.004275 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:31.004252 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istio-data\") pod \"e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:31.004352 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:31.004332 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istiod-ca-cert\") pod \"e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:31.005756 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:31.005734 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istio-envoy\") pod \"e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:31.005937 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:31.005918 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istio-podinfo\") pod \"e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:31.012940 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:31.012885 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istio-token\") pod \"e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:31.013454 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:31.013418 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdr5v\" (UniqueName: \"kubernetes.io/projected/1e4d01cd-4ac8-46af-bcc9-57708025ff82-kube-api-access-jdr5v\") pod \"e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:31.131398 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:31.131368 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-781d8b17-gw-openshift-default-dockercfg-rbgnp\"" Jul 24 03:48:31.139469 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:31.139436 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:31.283745 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:31.283718 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp"] Jul 24 03:48:31.284961 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:48:31.284935 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e4d01cd_4ac8_46af_bcc9_57708025ff82.slice/crio-cabf687fbe911032c40a7d613367dd7f19fafabfa10542ba043a0d14899d7374 WatchSource:0}: Error finding container cabf687fbe911032c40a7d613367dd7f19fafabfa10542ba043a0d14899d7374: Status 404 returned error can't find the container with id cabf687fbe911032c40a7d613367dd7f19fafabfa10542ba043a0d14899d7374 Jul 24 03:48:31.287113 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:31.287077 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:48:31.287216 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:31.287158 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:48:31.287216 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:31.287198 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:48:31.752857 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:31.752820 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" event={"ID":"1e4d01cd-4ac8-46af-bcc9-57708025ff82","Type":"ContainerStarted","Data":"ab1457a8ac41d7192f948c03e6be25c5369a1bc2396ccccfd074494b0210cdfe"} Jul 24 03:48:31.753061 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:31.752890 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" event={"ID":"1e4d01cd-4ac8-46af-bcc9-57708025ff82","Type":"ContainerStarted","Data":"cabf687fbe911032c40a7d613367dd7f19fafabfa10542ba043a0d14899d7374"} Jul 24 03:48:32.140328 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:32.140288 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:32.145142 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:32.145118 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:32.173759 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:32.173705 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" podStartSLOduration=2.173689612 podStartE2EDuration="2.173689612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:48:31.794495688 +0000 UTC m=+1861.233915144" watchObservedRunningTime="2026-07-24 03:48:32.173689612 +0000 UTC m=+1861.613109072" Jul 24 03:48:32.758532 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:32.758505 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:32.759637 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:32.759617 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:33.305760 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:33.305731 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:48:33.411340 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:33.411307 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7"] Jul 24 03:48:33.414874 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:33.414854 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7" Jul 24 03:48:33.420360 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:33.420341 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-781d8b17-serving-cert\"" Jul 24 03:48:33.431210 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:33.431188 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7"] Jul 24 03:48:33.527993 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:33.527958 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f62da9c2-1cee-428c-8548-8b77f98707ce-maas-api-tls\") pod \"maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7\" (UID: \"f62da9c2-1cee-428c-8548-8b77f98707ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7" Jul 24 03:48:33.528173 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:33.528013 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f62da9c2-1cee-428c-8548-8b77f98707ce-openshift-service-ca\") pod \"maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7\" (UID: \"f62da9c2-1cee-428c-8548-8b77f98707ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7" Jul 24 03:48:33.528173 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:33.528085 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6bbr\" (UniqueName: \"kubernetes.io/projected/f62da9c2-1cee-428c-8548-8b77f98707ce-kube-api-access-b6bbr\") pod \"maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7\" (UID: \"f62da9c2-1cee-428c-8548-8b77f98707ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7" Jul 24 03:48:33.629431 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:33.629400 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6bbr\" (UniqueName: \"kubernetes.io/projected/f62da9c2-1cee-428c-8548-8b77f98707ce-kube-api-access-b6bbr\") pod \"maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7\" (UID: \"f62da9c2-1cee-428c-8548-8b77f98707ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7" Jul 24 03:48:33.629636 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:33.629540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f62da9c2-1cee-428c-8548-8b77f98707ce-maas-api-tls\") pod \"maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7\" (UID: \"f62da9c2-1cee-428c-8548-8b77f98707ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7" Jul 24 03:48:33.629636 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:33.629579 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f62da9c2-1cee-428c-8548-8b77f98707ce-openshift-service-ca\") pod \"maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7\" (UID: \"f62da9c2-1cee-428c-8548-8b77f98707ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7" Jul 24 03:48:33.629747 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:48:33.629693 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-serving-cert: secret "maas-api-e2e-migrate-781d8b17-serving-cert" not found Jul 24 03:48:33.629803 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:48:33.629774 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f62da9c2-1cee-428c-8548-8b77f98707ce-maas-api-tls podName:f62da9c2-1cee-428c-8548-8b77f98707ce nodeName:}" failed. No retries permitted until 2026-07-24 03:48:34.129753902 +0000 UTC m=+1863.569173340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f62da9c2-1cee-428c-8548-8b77f98707ce-maas-api-tls") pod "maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7" (UID: "f62da9c2-1cee-428c-8548-8b77f98707ce") : secret "maas-api-e2e-migrate-781d8b17-serving-cert" not found Jul 24 03:48:33.630253 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:33.630235 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f62da9c2-1cee-428c-8548-8b77f98707ce-openshift-service-ca\") pod \"maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7\" (UID: \"f62da9c2-1cee-428c-8548-8b77f98707ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7" Jul 24 03:48:33.638567 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:33.638540 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6bbr\" (UniqueName: \"kubernetes.io/projected/f62da9c2-1cee-428c-8548-8b77f98707ce-kube-api-access-b6bbr\") pod \"maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7\" (UID: \"f62da9c2-1cee-428c-8548-8b77f98707ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7" Jul 24 03:48:34.135517 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:34.135455 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f62da9c2-1cee-428c-8548-8b77f98707ce-maas-api-tls\") pod \"maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7\" (UID: \"f62da9c2-1cee-428c-8548-8b77f98707ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7" Jul 24 03:48:34.139309 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:34.139277 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f62da9c2-1cee-428c-8548-8b77f98707ce-maas-api-tls\") pod \"maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7\" (UID: \"f62da9c2-1cee-428c-8548-8b77f98707ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7" Jul 24 03:48:34.326047 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:34.326010 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7" Jul 24 03:48:34.497668 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:34.497644 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7"] Jul 24 03:48:34.499324 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:48:34.499292 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf62da9c2_1cee_428c_8548_8b77f98707ce.slice/crio-2dbe6a5dd939c5bcba9539daa70cd5491f31cde4f030df267d1196c3fb5496b9 WatchSource:0}: Error finding container 2dbe6a5dd939c5bcba9539daa70cd5491f31cde4f030df267d1196c3fb5496b9: Status 404 returned error can't find the container with id 2dbe6a5dd939c5bcba9539daa70cd5491f31cde4f030df267d1196c3fb5496b9 Jul 24 03:48:34.531948 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:34.531919 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:48:34.770244 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:34.770215 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7" event={"ID":"f62da9c2-1cee-428c-8548-8b77f98707ce","Type":"ContainerStarted","Data":"2dbe6a5dd939c5bcba9539daa70cd5491f31cde4f030df267d1196c3fb5496b9"} Jul 24 03:48:35.018296 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.018240 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:48:35.146679 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.146650 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-workload-socket\") pod \"e54f2362-710a-443b-b5b1-c47e529bfac1\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " Jul 24 03:48:35.146880 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.146713 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-workload-certs\") pod \"e54f2362-710a-443b-b5b1-c47e529bfac1\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " Jul 24 03:48:35.146880 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.146747 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-credential-socket\") pod \"e54f2362-710a-443b-b5b1-c47e529bfac1\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " Jul 24 03:48:35.146880 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.146812 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-istio-envoy\") pod \"e54f2362-710a-443b-b5b1-c47e529bfac1\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " Jul 24 03:48:35.146880 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.146867 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tf949\" (UniqueName: \"kubernetes.io/projected/e54f2362-710a-443b-b5b1-c47e529bfac1-kube-api-access-tf949\") pod \"e54f2362-710a-443b-b5b1-c47e529bfac1\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " Jul 24 03:48:35.147104 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.146890 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e54f2362-710a-443b-b5b1-c47e529bfac1-istio-token\") pod \"e54f2362-710a-443b-b5b1-c47e529bfac1\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " Jul 24 03:48:35.147104 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.146928 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-istio-data\") pod \"e54f2362-710a-443b-b5b1-c47e529bfac1\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " Jul 24 03:48:35.147104 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.146962 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e54f2362-710a-443b-b5b1-c47e529bfac1-istiod-ca-cert\") pod \"e54f2362-710a-443b-b5b1-c47e529bfac1\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " Jul 24 03:48:35.147104 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.147002 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e54f2362-710a-443b-b5b1-c47e529bfac1-istio-podinfo\") pod \"e54f2362-710a-443b-b5b1-c47e529bfac1\" (UID: \"e54f2362-710a-443b-b5b1-c47e529bfac1\") " Jul 24 03:48:35.147324 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.146953 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e54f2362-710a-443b-b5b1-c47e529bfac1" (UID: "e54f2362-710a-443b-b5b1-c47e529bfac1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:48:35.147324 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.147018 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e54f2362-710a-443b-b5b1-c47e529bfac1" (UID: "e54f2362-710a-443b-b5b1-c47e529bfac1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:48:35.147324 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.147119 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e54f2362-710a-443b-b5b1-c47e529bfac1" (UID: "e54f2362-710a-443b-b5b1-c47e529bfac1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:48:35.147324 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.147282 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e54f2362-710a-443b-b5b1-c47e529bfac1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e54f2362-710a-443b-b5b1-c47e529bfac1" (UID: "e54f2362-710a-443b-b5b1-c47e529bfac1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 03:48:35.147555 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.147500 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-workload-certs\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:48:35.147555 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.147521 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-credential-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:48:35.147663 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.147593 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e54f2362-710a-443b-b5b1-c47e529bfac1-istiod-ca-cert\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:48:35.147858 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.147712 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-workload-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:48:35.147979 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.147951 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-istio-data" (OuterVolumeSpecName: "istio-data") pod "e54f2362-710a-443b-b5b1-c47e529bfac1" (UID: "e54f2362-710a-443b-b5b1-c47e529bfac1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:48:35.149196 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.149167 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54f2362-710a-443b-b5b1-c47e529bfac1-kube-api-access-tf949" (OuterVolumeSpecName: "kube-api-access-tf949") pod "e54f2362-710a-443b-b5b1-c47e529bfac1" (UID: "e54f2362-710a-443b-b5b1-c47e529bfac1"). InnerVolumeSpecName "kube-api-access-tf949". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:48:35.149327 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.149276 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e54f2362-710a-443b-b5b1-c47e529bfac1" (UID: "e54f2362-710a-443b-b5b1-c47e529bfac1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:48:35.149327 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.149278 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e54f2362-710a-443b-b5b1-c47e529bfac1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e54f2362-710a-443b-b5b1-c47e529bfac1" (UID: "e54f2362-710a-443b-b5b1-c47e529bfac1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 03:48:35.149514 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.149493 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54f2362-710a-443b-b5b1-c47e529bfac1-istio-token" (OuterVolumeSpecName: "istio-token") pod "e54f2362-710a-443b-b5b1-c47e529bfac1" (UID: "e54f2362-710a-443b-b5b1-c47e529bfac1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:48:35.248771 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.248739 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf949\" (UniqueName: \"kubernetes.io/projected/e54f2362-710a-443b-b5b1-c47e529bfac1-kube-api-access-tf949\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:48:35.248771 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.248769 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e54f2362-710a-443b-b5b1-c47e529bfac1-istio-token\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:48:35.248771 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.248779 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-istio-data\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:48:35.248997 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.248787 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e54f2362-710a-443b-b5b1-c47e529bfac1-istio-podinfo\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:48:35.248997 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.248795 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e54f2362-710a-443b-b5b1-c47e529bfac1-istio-envoy\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:48:35.776660 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.776627 2579 generic.go:356] "Generic (PLEG): container finished" podID="e54f2362-710a-443b-b5b1-c47e529bfac1" containerID="051e83cf6fd8fc0711fe2f66aff2228d10d5b3aa530402d6ccbcfa0ccd90f973" exitCode=0 Jul 24 03:48:35.777055 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.776671 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" Jul 24 03:48:35.777055 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.776701 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" event={"ID":"e54f2362-710a-443b-b5b1-c47e529bfac1","Type":"ContainerDied","Data":"051e83cf6fd8fc0711fe2f66aff2228d10d5b3aa530402d6ccbcfa0ccd90f973"} Jul 24 03:48:35.777055 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.776762 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp" event={"ID":"e54f2362-710a-443b-b5b1-c47e529bfac1","Type":"ContainerDied","Data":"b43385906b64e095739de091208a638c556e314ecb1355a9f6f55aa32dd5c1c2"} Jul 24 03:48:35.777055 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.776798 2579 scope.go:117] "RemoveContainer" containerID="051e83cf6fd8fc0711fe2f66aff2228d10d5b3aa530402d6ccbcfa0ccd90f973" Jul 24 03:48:35.778515 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.778492 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7" event={"ID":"f62da9c2-1cee-428c-8548-8b77f98707ce","Type":"ContainerStarted","Data":"10438b31f940f61317ef8490def551819578884013f5fc65a259f8587c4daf6e"} Jul 24 03:48:35.778654 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.778549 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7" Jul 24 03:48:35.786374 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.786356 2579 scope.go:117] "RemoveContainer" containerID="3c989dd846d6e79c5d50c29516f3f327f79256d1fef811d0ff4f1abe45a5ccc5" Jul 24 03:48:35.795138 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.795122 2579 scope.go:117] "RemoveContainer" containerID="051e83cf6fd8fc0711fe2f66aff2228d10d5b3aa530402d6ccbcfa0ccd90f973" Jul 24 03:48:35.795397 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:48:35.795379 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"051e83cf6fd8fc0711fe2f66aff2228d10d5b3aa530402d6ccbcfa0ccd90f973\": container with ID starting with 051e83cf6fd8fc0711fe2f66aff2228d10d5b3aa530402d6ccbcfa0ccd90f973 not found: ID does not exist" containerID="051e83cf6fd8fc0711fe2f66aff2228d10d5b3aa530402d6ccbcfa0ccd90f973" Jul 24 03:48:35.795443 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.795404 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"051e83cf6fd8fc0711fe2f66aff2228d10d5b3aa530402d6ccbcfa0ccd90f973"} err="failed to get container status \"051e83cf6fd8fc0711fe2f66aff2228d10d5b3aa530402d6ccbcfa0ccd90f973\": rpc error: code = NotFound desc = could not find container \"051e83cf6fd8fc0711fe2f66aff2228d10d5b3aa530402d6ccbcfa0ccd90f973\": container with ID starting with 051e83cf6fd8fc0711fe2f66aff2228d10d5b3aa530402d6ccbcfa0ccd90f973 not found: ID does not exist" Jul 24 03:48:35.795443 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.795421 2579 scope.go:117] "RemoveContainer" containerID="3c989dd846d6e79c5d50c29516f3f327f79256d1fef811d0ff4f1abe45a5ccc5" Jul 24 03:48:35.795641 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:48:35.795627 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c989dd846d6e79c5d50c29516f3f327f79256d1fef811d0ff4f1abe45a5ccc5\": container with ID starting with 3c989dd846d6e79c5d50c29516f3f327f79256d1fef811d0ff4f1abe45a5ccc5 not found: ID does not exist" containerID="3c989dd846d6e79c5d50c29516f3f327f79256d1fef811d0ff4f1abe45a5ccc5" Jul 24 03:48:35.795679 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.795645 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c989dd846d6e79c5d50c29516f3f327f79256d1fef811d0ff4f1abe45a5ccc5"} err="failed to get container status \"3c989dd846d6e79c5d50c29516f3f327f79256d1fef811d0ff4f1abe45a5ccc5\": rpc error: code = NotFound desc = could not find container \"3c989dd846d6e79c5d50c29516f3f327f79256d1fef811d0ff4f1abe45a5ccc5\": container with ID starting with 3c989dd846d6e79c5d50c29516f3f327f79256d1fef811d0ff4f1abe45a5ccc5 not found: ID does not exist" Jul 24 03:48:35.798604 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.798564 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7" podStartSLOduration=2.492051948 podStartE2EDuration="2.798551419s" totalImagesPullingTime="306.499471ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:48:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:48:35.796296535 +0000 UTC m=+1865.235715995" watchObservedRunningTime="2026-07-24 03:48:35.798551419 +0000 UTC m=+1865.237970929" Jul 24 03:48:35.809132 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.809108 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp"] Jul 24 03:48:35.812402 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:35.812382 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3b0f86fb-openshift-default-6694dd778b-89dqp"] Jul 24 03:48:37.037491 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:37.036203 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e54f2362-710a-443b-b5b1-c47e529bfac1" path="/var/lib/kubelet/pods/e54f2362-710a-443b-b5b1-c47e529bfac1/volumes" Jul 24 03:48:41.797936 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:41.797894 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7" Jul 24 03:48:47.760357 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:47.760276 2579 patch_prober.go:28] interesting pod/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 24 03:48:47.760792 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:47.760359 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" podUID="1e4d01cd-4ac8-46af-bcc9-57708025ff82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 24 03:48:47.844421 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:47.844330 2579 generic.go:356] "Generic (PLEG): container finished" podID="1e4d01cd-4ac8-46af-bcc9-57708025ff82" containerID="ab1457a8ac41d7192f948c03e6be25c5369a1bc2396ccccfd074494b0210cdfe" exitCode=0 Jul 24 03:48:47.844421 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:47.844369 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" event={"ID":"1e4d01cd-4ac8-46af-bcc9-57708025ff82","Type":"ContainerDied","Data":"ab1457a8ac41d7192f948c03e6be25c5369a1bc2396ccccfd074494b0210cdfe"} Jul 24 03:48:47.844813 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:47.844792 2579 scope.go:117] "RemoveContainer" containerID="ab1457a8ac41d7192f948c03e6be25c5369a1bc2396ccccfd074494b0210cdfe" Jul 24 03:48:47.845849 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:47.845819 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:48:47.845900 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:47.845887 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:48:47.845940 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:47.845919 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:48:48.851355 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:48.851316 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" event={"ID":"1e4d01cd-4ac8-46af-bcc9-57708025ff82","Type":"ContainerStarted","Data":"779bfa91685e9d210449cab2e7152aa2150c35264d6cb6a9e557023a247827f5"} Jul 24 03:48:49.140633 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:49.140601 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:49.140840 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:49.140818 2579 patch_prober.go:28] interesting pod/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 24 03:48:49.140891 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:49.140866 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" podUID="1e4d01cd-4ac8-46af-bcc9-57708025ff82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 24 03:48:50.140728 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:50.140684 2579 patch_prober.go:28] interesting pod/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 24 03:48:50.141098 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:50.140748 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" podUID="1e4d01cd-4ac8-46af-bcc9-57708025ff82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 24 03:48:51.140712 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:51.140671 2579 patch_prober.go:28] interesting pod/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 24 03:48:51.140906 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:51.140734 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" podUID="1e4d01cd-4ac8-46af-bcc9-57708025ff82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 24 03:48:52.140852 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:52.140766 2579 patch_prober.go:28] interesting pod/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 24 03:48:52.140852 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:52.140835 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" podUID="1e4d01cd-4ac8-46af-bcc9-57708025ff82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 24 03:48:53.144944 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:53.144913 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:53.145422 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:53.145286 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:48:53.145858 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:48:53.145839 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:49:08.916374 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:08.916339 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7"] Jul 24 03:49:08.917275 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:08.917240 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7" podUID="f62da9c2-1cee-428c-8548-8b77f98707ce" containerName="maas-api" containerID="cri-o://10438b31f940f61317ef8490def551819578884013f5fc65a259f8587c4daf6e" gracePeriod=30 Jul 24 03:49:08.944673 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:08.944648 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:49:09.007675 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:09.007637 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:49:09.077860 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:09.077812 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:49:09.184070 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:09.184045 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7" Jul 24 03:49:09.292952 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:09.292917 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6bbr\" (UniqueName: \"kubernetes.io/projected/f62da9c2-1cee-428c-8548-8b77f98707ce-kube-api-access-b6bbr\") pod \"f62da9c2-1cee-428c-8548-8b77f98707ce\" (UID: \"f62da9c2-1cee-428c-8548-8b77f98707ce\") " Jul 24 03:49:09.293122 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:09.292995 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f62da9c2-1cee-428c-8548-8b77f98707ce-openshift-service-ca\") pod \"f62da9c2-1cee-428c-8548-8b77f98707ce\" (UID: \"f62da9c2-1cee-428c-8548-8b77f98707ce\") " Jul 24 03:49:09.293122 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:09.293059 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f62da9c2-1cee-428c-8548-8b77f98707ce-maas-api-tls\") pod \"f62da9c2-1cee-428c-8548-8b77f98707ce\" (UID: \"f62da9c2-1cee-428c-8548-8b77f98707ce\") " Jul 24 03:49:09.293415 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:09.293386 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f62da9c2-1cee-428c-8548-8b77f98707ce-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f62da9c2-1cee-428c-8548-8b77f98707ce" (UID: "f62da9c2-1cee-428c-8548-8b77f98707ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 03:49:09.295233 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:09.295197 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f62da9c2-1cee-428c-8548-8b77f98707ce-kube-api-access-b6bbr" (OuterVolumeSpecName: "kube-api-access-b6bbr") pod "f62da9c2-1cee-428c-8548-8b77f98707ce" (UID: "f62da9c2-1cee-428c-8548-8b77f98707ce"). InnerVolumeSpecName "kube-api-access-b6bbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:49:09.295330 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:09.295269 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f62da9c2-1cee-428c-8548-8b77f98707ce-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f62da9c2-1cee-428c-8548-8b77f98707ce" (UID: "f62da9c2-1cee-428c-8548-8b77f98707ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 03:49:09.394431 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:09.394391 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f62da9c2-1cee-428c-8548-8b77f98707ce-openshift-service-ca\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:49:09.394431 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:09.394426 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f62da9c2-1cee-428c-8548-8b77f98707ce-maas-api-tls\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:49:09.394431 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:09.394436 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6bbr\" (UniqueName: \"kubernetes.io/projected/f62da9c2-1cee-428c-8548-8b77f98707ce-kube-api-access-b6bbr\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:49:09.966035 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:09.965998 2579 generic.go:356] "Generic (PLEG): container finished" podID="f62da9c2-1cee-428c-8548-8b77f98707ce" containerID="10438b31f940f61317ef8490def551819578884013f5fc65a259f8587c4daf6e" exitCode=0 Jul 24 03:49:09.966533 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:09.966043 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7" Jul 24 03:49:09.966533 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:09.966086 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7" event={"ID":"f62da9c2-1cee-428c-8548-8b77f98707ce","Type":"ContainerDied","Data":"10438b31f940f61317ef8490def551819578884013f5fc65a259f8587c4daf6e"} Jul 24 03:49:09.966533 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:09.966179 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7" event={"ID":"f62da9c2-1cee-428c-8548-8b77f98707ce","Type":"ContainerDied","Data":"2dbe6a5dd939c5bcba9539daa70cd5491f31cde4f030df267d1196c3fb5496b9"} Jul 24 03:49:09.966533 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:09.966213 2579 scope.go:117] "RemoveContainer" containerID="10438b31f940f61317ef8490def551819578884013f5fc65a259f8587c4daf6e" Jul 24 03:49:09.975242 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:09.975226 2579 scope.go:117] "RemoveContainer" containerID="10438b31f940f61317ef8490def551819578884013f5fc65a259f8587c4daf6e" Jul 24 03:49:09.975506 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:49:09.975489 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10438b31f940f61317ef8490def551819578884013f5fc65a259f8587c4daf6e\": container with ID starting with 10438b31f940f61317ef8490def551819578884013f5fc65a259f8587c4daf6e not found: ID does not exist" containerID="10438b31f940f61317ef8490def551819578884013f5fc65a259f8587c4daf6e" Jul 24 03:49:09.975555 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:09.975515 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10438b31f940f61317ef8490def551819578884013f5fc65a259f8587c4daf6e"} err="failed to get container status \"10438b31f940f61317ef8490def551819578884013f5fc65a259f8587c4daf6e\": rpc error: code = NotFound desc = could not find container \"10438b31f940f61317ef8490def551819578884013f5fc65a259f8587c4daf6e\": container with ID starting with 10438b31f940f61317ef8490def551819578884013f5fc65a259f8587c4daf6e not found: ID does not exist" Jul 24 03:49:09.988639 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:09.988617 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7"] Jul 24 03:49:09.992696 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:09.992670 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-781d8b17-66cfdc5f66-8cbx7"] Jul 24 03:49:11.033354 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:11.033326 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f62da9c2-1cee-428c-8548-8b77f98707ce" path="/var/lib/kubelet/pods/f62da9c2-1cee-428c-8548-8b77f98707ce/volumes" Jul 24 03:49:13.471022 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:13.470988 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:49:13.609970 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:13.609936 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp"] Jul 24 03:49:13.610177 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:13.610153 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" podUID="1e4d01cd-4ac8-46af-bcc9-57708025ff82" containerName="istio-proxy" containerID="cri-o://779bfa91685e9d210449cab2e7152aa2150c35264d6cb6a9e557023a247827f5" gracePeriod=30 Jul 24 03:49:13.629368 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:13.629339 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:49:13.873387 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:13.873342 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:49:14.128674 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.128581 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:49:14.426445 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.426376 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7"] Jul 24 03:49:14.426850 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.426828 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e54f2362-710a-443b-b5b1-c47e529bfac1" containerName="istio-proxy" Jul 24 03:49:14.426850 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.426846 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e54f2362-710a-443b-b5b1-c47e529bfac1" containerName="istio-proxy" Jul 24 03:49:14.427010 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.426862 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f62da9c2-1cee-428c-8548-8b77f98707ce" containerName="maas-api" Jul 24 03:49:14.427010 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.426868 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f62da9c2-1cee-428c-8548-8b77f98707ce" containerName="maas-api" Jul 24 03:49:14.427010 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.426940 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e54f2362-710a-443b-b5b1-c47e529bfac1" containerName="istio-proxy" Jul 24 03:49:14.427010 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.426949 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f62da9c2-1cee-428c-8548-8b77f98707ce" containerName="maas-api" Jul 24 03:49:14.431128 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.431111 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.433531 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.433511 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c4c112d8-openshift-default-dockercfg-5wvnq\"" Jul 24 03:49:14.441067 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.441046 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7"] Jul 24 03:49:14.446094 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.446068 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:49:14.546141 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.546108 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-workload-certs\") pod \"e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.546141 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.546146 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-istio-data\") pod \"e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.546569 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.546250 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-istio-envoy\") pod \"e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.546569 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.546287 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-workload-socket\") pod \"e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.546569 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.546341 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gcch\" (UniqueName: \"kubernetes.io/projected/4d52ad46-69f9-4167-9871-2530b387a632-kube-api-access-2gcch\") pod \"e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.546569 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.546385 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-credential-socket\") pod \"e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.546569 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.546414 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d52ad46-69f9-4167-9871-2530b387a632-istio-token\") pod \"e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.546569 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.546448 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d52ad46-69f9-4167-9871-2530b387a632-istio-podinfo\") pod \"e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.546569 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.546509 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d52ad46-69f9-4167-9871-2530b387a632-istiod-ca-cert\") pod \"e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.647052 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.647014 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d52ad46-69f9-4167-9871-2530b387a632-istiod-ca-cert\") pod \"e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.647226 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.647066 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-workload-certs\") pod \"e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.647226 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.647085 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-istio-data\") pod \"e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.647226 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.647137 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-istio-envoy\") pod \"e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.647226 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.647157 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-workload-socket\") pod \"e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.647226 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.647185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gcch\" (UniqueName: \"kubernetes.io/projected/4d52ad46-69f9-4167-9871-2530b387a632-kube-api-access-2gcch\") pod \"e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.647226 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.647210 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-credential-socket\") pod \"e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.647566 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.647300 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d52ad46-69f9-4167-9871-2530b387a632-istio-token\") pod \"e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.647566 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.647377 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d52ad46-69f9-4167-9871-2530b387a632-istio-podinfo\") pod \"e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.647566 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.647487 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-workload-certs\") pod \"e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.647566 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.647530 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-istio-data\") pod \"e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.647824 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.647801 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-workload-socket\") pod \"e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.647937 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.647847 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-credential-socket\") pod \"e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.647937 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.647886 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d52ad46-69f9-4167-9871-2530b387a632-istiod-ca-cert\") pod \"e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.649962 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.649933 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d52ad46-69f9-4167-9871-2530b387a632-istio-podinfo\") pod \"e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.650049 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.649967 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-istio-envoy\") pod \"e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.656110 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.656092 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d52ad46-69f9-4167-9871-2530b387a632-istio-token\") pod \"e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.656190 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.656155 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gcch\" (UniqueName: \"kubernetes.io/projected/4d52ad46-69f9-4167-9871-2530b387a632-kube-api-access-2gcch\") pod \"e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.743603 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.743519 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:14.825447 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.825417 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:49:14.893491 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.893444 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7"] Jul 24 03:49:14.895123 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:49:14.895094 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d52ad46_69f9_4167_9871_2530b387a632.slice/crio-8df71dfbf8b966d954338586b75d15f8f06f41e70ab5bf8d18dae9c0166f4684 WatchSource:0}: Error finding container 8df71dfbf8b966d954338586b75d15f8f06f41e70ab5bf8d18dae9c0166f4684: Status 404 returned error can't find the container with id 8df71dfbf8b966d954338586b75d15f8f06f41e70ab5bf8d18dae9c0166f4684 Jul 24 03:49:14.897253 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.897216 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:49:14.897352 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.897300 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:49:14.897352 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.897331 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:49:14.994864 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.994764 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" event={"ID":"4d52ad46-69f9-4167-9871-2530b387a632","Type":"ContainerStarted","Data":"ede3abcd4d2adf105fd7d92d97ae74927e0cc291ef30a9f26e7bbbb5c1327cd2"} Jul 24 03:49:14.994864 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:14.994831 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" event={"ID":"4d52ad46-69f9-4167-9871-2530b387a632","Type":"ContainerStarted","Data":"8df71dfbf8b966d954338586b75d15f8f06f41e70ab5bf8d18dae9c0166f4684"} Jul 24 03:49:15.014483 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:15.014406 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" podStartSLOduration=1.014391002 podStartE2EDuration="1.014391002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:49:15.012771507 +0000 UTC m=+1904.452190969" watchObservedRunningTime="2026-07-24 03:49:15.014391002 +0000 UTC m=+1904.453810461" Jul 24 03:49:15.744690 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:15.744657 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:15.749619 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:15.749595 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:15.999872 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:15.999790 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:16.000766 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:16.000745 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:16.368128 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:16.368094 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:49:16.443996 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:16.443963 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x"] Jul 24 03:49:16.444498 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:16.444481 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e54f2362-710a-443b-b5b1-c47e529bfac1" containerName="istio-proxy" Jul 24 03:49:16.444614 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:16.444501 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e54f2362-710a-443b-b5b1-c47e529bfac1" containerName="istio-proxy" Jul 24 03:49:16.444614 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:16.444602 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e54f2362-710a-443b-b5b1-c47e529bfac1" containerName="istio-proxy" Jul 24 03:49:16.447859 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:16.447839 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x" Jul 24 03:49:16.450561 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:16.450538 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c4c112d8-serving-cert\"" Jul 24 03:49:16.450987 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:16.450956 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:49:16.459338 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:16.459191 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x"] Jul 24 03:49:16.489124 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:16.489087 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8sb6\" (UniqueName: \"kubernetes.io/projected/6d8b795e-518f-4834-b130-66f473ba4941-kube-api-access-x8sb6\") pod \"maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x\" (UID: \"6d8b795e-518f-4834-b130-66f473ba4941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x" Jul 24 03:49:16.489433 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:16.489414 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d8b795e-518f-4834-b130-66f473ba4941-maas-api-tls\") pod \"maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x\" (UID: \"6d8b795e-518f-4834-b130-66f473ba4941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x" Jul 24 03:49:16.489566 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:16.489552 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8b795e-518f-4834-b130-66f473ba4941-openshift-service-ca\") pod \"maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x\" (UID: \"6d8b795e-518f-4834-b130-66f473ba4941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x" Jul 24 03:49:16.516056 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:16.516026 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:49:16.590186 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:16.590152 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8sb6\" (UniqueName: \"kubernetes.io/projected/6d8b795e-518f-4834-b130-66f473ba4941-kube-api-access-x8sb6\") pod \"maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x\" (UID: \"6d8b795e-518f-4834-b130-66f473ba4941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x" Jul 24 03:49:16.590354 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:16.590224 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d8b795e-518f-4834-b130-66f473ba4941-maas-api-tls\") pod \"maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x\" (UID: \"6d8b795e-518f-4834-b130-66f473ba4941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x" Jul 24 03:49:16.590354 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:16.590245 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8b795e-518f-4834-b130-66f473ba4941-openshift-service-ca\") pod \"maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x\" (UID: \"6d8b795e-518f-4834-b130-66f473ba4941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x" Jul 24 03:49:16.590434 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:49:16.590390 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-serving-cert: secret "maas-api-e2e-ait-c4c112d8-serving-cert" not found Jul 24 03:49:16.590506 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:49:16.590496 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d8b795e-518f-4834-b130-66f473ba4941-maas-api-tls podName:6d8b795e-518f-4834-b130-66f473ba4941 nodeName:}" failed. No retries permitted until 2026-07-24 03:49:17.090453082 +0000 UTC m=+1906.529872524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6d8b795e-518f-4834-b130-66f473ba4941-maas-api-tls") pod "maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x" (UID: "6d8b795e-518f-4834-b130-66f473ba4941") : secret "maas-api-e2e-ait-c4c112d8-serving-cert" not found Jul 24 03:49:16.590910 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:16.590893 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8b795e-518f-4834-b130-66f473ba4941-openshift-service-ca\") pod \"maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x\" (UID: \"6d8b795e-518f-4834-b130-66f473ba4941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x" Jul 24 03:49:16.598803 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:16.598780 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8sb6\" (UniqueName: \"kubernetes.io/projected/6d8b795e-518f-4834-b130-66f473ba4941-kube-api-access-x8sb6\") pod \"maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x\" (UID: \"6d8b795e-518f-4834-b130-66f473ba4941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x" Jul 24 03:49:16.737432 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:16.737339 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:49:17.015541 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:17.015436 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:49:17.094577 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:17.094544 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d8b795e-518f-4834-b130-66f473ba4941-maas-api-tls\") pod \"maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x\" (UID: \"6d8b795e-518f-4834-b130-66f473ba4941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x" Jul 24 03:49:17.097107 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:17.097084 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d8b795e-518f-4834-b130-66f473ba4941-maas-api-tls\") pod \"maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x\" (UID: \"6d8b795e-518f-4834-b130-66f473ba4941\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x" Jul 24 03:49:17.116776 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:17.116752 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:49:17.388443 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:17.388405 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x" Jul 24 03:49:17.533925 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:17.533897 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x"] Jul 24 03:49:17.535498 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:49:17.535476 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8b795e_518f_4834_b130_66f473ba4941.slice/crio-42e22ca15466cae2d685086aa280b1099f14b0dd6e25c70072896320e3945cb9 WatchSource:0}: Error finding container 42e22ca15466cae2d685086aa280b1099f14b0dd6e25c70072896320e3945cb9: Status 404 returned error can't find the container with id 42e22ca15466cae2d685086aa280b1099f14b0dd6e25c70072896320e3945cb9 Jul 24 03:49:18.016969 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:18.016934 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x" event={"ID":"6d8b795e-518f-4834-b130-66f473ba4941","Type":"ContainerStarted","Data":"2f1578346ed09e0a599ada6679239b8e77dc3745f842936fff25e064ed90f511"} Jul 24 03:49:18.017501 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:18.017454 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x" Jul 24 03:49:18.017615 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:18.017523 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x" event={"ID":"6d8b795e-518f-4834-b130-66f473ba4941","Type":"ContainerStarted","Data":"42e22ca15466cae2d685086aa280b1099f14b0dd6e25c70072896320e3945cb9"} Jul 24 03:49:18.032925 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:18.032881 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x" podStartSLOduration=1.629622923 podStartE2EDuration="2.032866702s" totalImagesPullingTime="403.243779ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:49:18.032546416 +0000 UTC m=+1907.471965887" watchObservedRunningTime="2026-07-24 03:49:18.032866702 +0000 UTC m=+1907.472286163" Jul 24 03:49:18.862281 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:18.862253 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:49:18.910140 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:18.910050 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istio-token\") pod \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " Jul 24 03:49:18.910140 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:18.910087 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdr5v\" (UniqueName: \"kubernetes.io/projected/1e4d01cd-4ac8-46af-bcc9-57708025ff82-kube-api-access-jdr5v\") pod \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " Jul 24 03:49:18.910140 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:18.910107 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istio-envoy\") pod \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " Jul 24 03:49:18.910490 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:18.910437 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-workload-socket\") pod \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " Jul 24 03:49:18.910569 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:18.910512 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-workload-certs\") pod \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " Jul 24 03:49:18.910629 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:18.910583 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-credential-socket\") pod \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " Jul 24 03:49:18.910690 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:18.910625 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istio-podinfo\") pod \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " Jul 24 03:49:18.910690 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:18.910652 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istiod-ca-cert\") pod \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " Jul 24 03:49:18.910690 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:18.910676 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istio-data\") pod \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\" (UID: \"1e4d01cd-4ac8-46af-bcc9-57708025ff82\") " Jul 24 03:49:18.911130 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:18.911103 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1e4d01cd-4ac8-46af-bcc9-57708025ff82" (UID: "1e4d01cd-4ac8-46af-bcc9-57708025ff82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:49:18.911769 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:18.911360 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1e4d01cd-4ac8-46af-bcc9-57708025ff82" (UID: "1e4d01cd-4ac8-46af-bcc9-57708025ff82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:49:18.911769 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:18.911608 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1e4d01cd-4ac8-46af-bcc9-57708025ff82" (UID: "1e4d01cd-4ac8-46af-bcc9-57708025ff82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:49:18.911769 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:18.911718 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1e4d01cd-4ac8-46af-bcc9-57708025ff82" (UID: "1e4d01cd-4ac8-46af-bcc9-57708025ff82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 03:49:18.912017 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:18.911878 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istio-data" (OuterVolumeSpecName: "istio-data") pod "1e4d01cd-4ac8-46af-bcc9-57708025ff82" (UID: "1e4d01cd-4ac8-46af-bcc9-57708025ff82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:49:18.912930 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:18.912908 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e4d01cd-4ac8-46af-bcc9-57708025ff82-kube-api-access-jdr5v" (OuterVolumeSpecName: "kube-api-access-jdr5v") pod "1e4d01cd-4ac8-46af-bcc9-57708025ff82" (UID: "1e4d01cd-4ac8-46af-bcc9-57708025ff82"). InnerVolumeSpecName "kube-api-access-jdr5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:49:18.913089 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:18.912967 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1e4d01cd-4ac8-46af-bcc9-57708025ff82" (UID: "1e4d01cd-4ac8-46af-bcc9-57708025ff82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:49:18.913831 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:18.913804 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istio-token" (OuterVolumeSpecName: "istio-token") pod "1e4d01cd-4ac8-46af-bcc9-57708025ff82" (UID: "1e4d01cd-4ac8-46af-bcc9-57708025ff82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:49:18.914175 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:18.914155 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1e4d01cd-4ac8-46af-bcc9-57708025ff82" (UID: "1e4d01cd-4ac8-46af-bcc9-57708025ff82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 03:49:19.011752 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:19.011724 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-credential-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:49:19.011752 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:19.011750 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istio-podinfo\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:49:19.011752 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:19.011759 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istiod-ca-cert\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:49:19.011944 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:19.011767 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istio-data\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:49:19.011944 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:19.011775 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istio-token\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:49:19.011944 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:19.011786 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdr5v\" (UniqueName: \"kubernetes.io/projected/1e4d01cd-4ac8-46af-bcc9-57708025ff82-kube-api-access-jdr5v\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:49:19.011944 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:19.011794 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-istio-envoy\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:49:19.011944 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:19.011802 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-workload-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:49:19.011944 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:19.011809 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e4d01cd-4ac8-46af-bcc9-57708025ff82-workload-certs\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:49:19.024648 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:19.024619 2579 generic.go:356] "Generic (PLEG): container finished" podID="1e4d01cd-4ac8-46af-bcc9-57708025ff82" containerID="779bfa91685e9d210449cab2e7152aa2150c35264d6cb6a9e557023a247827f5" exitCode=0 Jul 24 03:49:19.025021 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:19.024677 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" Jul 24 03:49:19.025021 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:19.024704 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" event={"ID":"1e4d01cd-4ac8-46af-bcc9-57708025ff82","Type":"ContainerDied","Data":"779bfa91685e9d210449cab2e7152aa2150c35264d6cb6a9e557023a247827f5"} Jul 24 03:49:19.025021 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:19.024759 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp" event={"ID":"1e4d01cd-4ac8-46af-bcc9-57708025ff82","Type":"ContainerDied","Data":"cabf687fbe911032c40a7d613367dd7f19fafabfa10542ba043a0d14899d7374"} Jul 24 03:49:19.025021 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:19.024781 2579 scope.go:117] "RemoveContainer" containerID="779bfa91685e9d210449cab2e7152aa2150c35264d6cb6a9e557023a247827f5" Jul 24 03:49:19.034951 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:19.034934 2579 scope.go:117] "RemoveContainer" containerID="ab1457a8ac41d7192f948c03e6be25c5369a1bc2396ccccfd074494b0210cdfe" Jul 24 03:49:19.043071 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:19.043053 2579 scope.go:117] "RemoveContainer" containerID="779bfa91685e9d210449cab2e7152aa2150c35264d6cb6a9e557023a247827f5" Jul 24 03:49:19.043299 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:49:19.043280 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"779bfa91685e9d210449cab2e7152aa2150c35264d6cb6a9e557023a247827f5\": container with ID starting with 779bfa91685e9d210449cab2e7152aa2150c35264d6cb6a9e557023a247827f5 not found: ID does not exist" containerID="779bfa91685e9d210449cab2e7152aa2150c35264d6cb6a9e557023a247827f5" Jul 24 03:49:19.043349 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:19.043308 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779bfa91685e9d210449cab2e7152aa2150c35264d6cb6a9e557023a247827f5"} err="failed to get container status \"779bfa91685e9d210449cab2e7152aa2150c35264d6cb6a9e557023a247827f5\": rpc error: code = NotFound desc = could not find container \"779bfa91685e9d210449cab2e7152aa2150c35264d6cb6a9e557023a247827f5\": container with ID starting with 779bfa91685e9d210449cab2e7152aa2150c35264d6cb6a9e557023a247827f5 not found: ID does not exist" Jul 24 03:49:19.043349 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:19.043324 2579 scope.go:117] "RemoveContainer" containerID="ab1457a8ac41d7192f948c03e6be25c5369a1bc2396ccccfd074494b0210cdfe" Jul 24 03:49:19.043593 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:49:19.043574 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab1457a8ac41d7192f948c03e6be25c5369a1bc2396ccccfd074494b0210cdfe\": container with ID starting with ab1457a8ac41d7192f948c03e6be25c5369a1bc2396ccccfd074494b0210cdfe not found: ID does not exist" containerID="ab1457a8ac41d7192f948c03e6be25c5369a1bc2396ccccfd074494b0210cdfe" Jul 24 03:49:19.043655 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:19.043598 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab1457a8ac41d7192f948c03e6be25c5369a1bc2396ccccfd074494b0210cdfe"} err="failed to get container status \"ab1457a8ac41d7192f948c03e6be25c5369a1bc2396ccccfd074494b0210cdfe\": rpc error: code = NotFound desc = could not find container \"ab1457a8ac41d7192f948c03e6be25c5369a1bc2396ccccfd074494b0210cdfe\": container with ID starting with ab1457a8ac41d7192f948c03e6be25c5369a1bc2396ccccfd074494b0210cdfe not found: ID does not exist" Jul 24 03:49:19.047408 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:19.047386 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp"] Jul 24 03:49:19.056061 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:19.056042 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-781d8b17-gw-openshift-default-557694965b-gc7mp"] Jul 24 03:49:21.034527 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:21.034497 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e4d01cd-4ac8-46af-bcc9-57708025ff82" path="/var/lib/kubelet/pods/1e4d01cd-4ac8-46af-bcc9-57708025ff82/volumes" Jul 24 03:49:25.039545 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:25.039518 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x" Jul 24 03:49:31.165787 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:31.164410 2579 patch_prober.go:28] interesting pod/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": read tcp 10.134.0.2:51284->10.134.0.54:15021: read: connection reset by peer" start-of-body= Jul 24 03:49:31.165787 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:31.164504 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" podUID="4d52ad46-69f9-4167-9871-2530b387a632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": read tcp 10.134.0.2:51284->10.134.0.54:15021: read: connection reset by peer" Jul 24 03:49:32.092162 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:32.092132 2579 generic.go:356] "Generic (PLEG): container finished" podID="4d52ad46-69f9-4167-9871-2530b387a632" containerID="ede3abcd4d2adf105fd7d92d97ae74927e0cc291ef30a9f26e7bbbb5c1327cd2" exitCode=0 Jul 24 03:49:32.092341 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:32.092210 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" event={"ID":"4d52ad46-69f9-4167-9871-2530b387a632","Type":"ContainerDied","Data":"ede3abcd4d2adf105fd7d92d97ae74927e0cc291ef30a9f26e7bbbb5c1327cd2"} Jul 24 03:49:32.092653 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:32.092639 2579 scope.go:117] "RemoveContainer" containerID="ede3abcd4d2adf105fd7d92d97ae74927e0cc291ef30a9f26e7bbbb5c1327cd2" Jul 24 03:49:32.093784 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:32.093756 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:49:32.093852 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:32.093825 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:49:32.093889 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:32.093855 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:49:33.101629 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:33.101597 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" event={"ID":"4d52ad46-69f9-4167-9871-2530b387a632","Type":"ContainerStarted","Data":"9ecf3baebdd9a4b5a42f3c85c737e88d21ba3cd09fa20f6c39e75486a33acf15"} Jul 24 03:49:33.744640 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:33.744605 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:33.744848 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:33.744811 2579 patch_prober.go:28] interesting pod/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 24 03:49:33.744923 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:33.744867 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" podUID="4d52ad46-69f9-4167-9871-2530b387a632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 24 03:49:34.744775 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:34.744739 2579 patch_prober.go:28] interesting pod/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 24 03:49:34.745165 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:34.744798 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" podUID="4d52ad46-69f9-4167-9871-2530b387a632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 24 03:49:35.744644 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:35.744596 2579 patch_prober.go:28] interesting pod/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 24 03:49:35.744830 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:35.744665 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" podUID="4d52ad46-69f9-4167-9871-2530b387a632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 24 03:49:36.744757 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:36.744718 2579 patch_prober.go:28] interesting pod/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 24 03:49:36.744944 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:36.744775 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" podUID="4d52ad46-69f9-4167-9871-2530b387a632" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 24 03:49:37.747642 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:37.747611 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:37.748023 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:37.747858 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:37.748613 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:37.748593 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:49:52.592672 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:52.592632 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x"] Jul 24 03:49:52.593163 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:52.592919 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x" podUID="6d8b795e-518f-4834-b130-66f473ba4941" containerName="maas-api" containerID="cri-o://2f1578346ed09e0a599ada6679239b8e77dc3745f842936fff25e064ed90f511" gracePeriod=30 Jul 24 03:49:52.625644 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:52.625619 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:49:52.726361 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:52.726325 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:49:52.829169 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:52.829087 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:49:52.858724 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:52.858702 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x" Jul 24 03:49:52.960962 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:52.960932 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8b795e-518f-4834-b130-66f473ba4941-openshift-service-ca\") pod \"6d8b795e-518f-4834-b130-66f473ba4941\" (UID: \"6d8b795e-518f-4834-b130-66f473ba4941\") " Jul 24 03:49:52.961135 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:52.960985 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8sb6\" (UniqueName: \"kubernetes.io/projected/6d8b795e-518f-4834-b130-66f473ba4941-kube-api-access-x8sb6\") pod \"6d8b795e-518f-4834-b130-66f473ba4941\" (UID: \"6d8b795e-518f-4834-b130-66f473ba4941\") " Jul 24 03:49:52.961135 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:52.961039 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d8b795e-518f-4834-b130-66f473ba4941-maas-api-tls\") pod \"6d8b795e-518f-4834-b130-66f473ba4941\" (UID: \"6d8b795e-518f-4834-b130-66f473ba4941\") " Jul 24 03:49:52.961303 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:52.961279 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d8b795e-518f-4834-b130-66f473ba4941-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6d8b795e-518f-4834-b130-66f473ba4941" (UID: "6d8b795e-518f-4834-b130-66f473ba4941"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 03:49:52.961419 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:52.961399 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8b795e-518f-4834-b130-66f473ba4941-openshift-service-ca\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:49:52.963279 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:52.963259 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8b795e-518f-4834-b130-66f473ba4941-kube-api-access-x8sb6" (OuterVolumeSpecName: "kube-api-access-x8sb6") pod "6d8b795e-518f-4834-b130-66f473ba4941" (UID: "6d8b795e-518f-4834-b130-66f473ba4941"). InnerVolumeSpecName "kube-api-access-x8sb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:49:52.963343 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:52.963321 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d8b795e-518f-4834-b130-66f473ba4941-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6d8b795e-518f-4834-b130-66f473ba4941" (UID: "6d8b795e-518f-4834-b130-66f473ba4941"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 03:49:53.062867 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:53.062840 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8sb6\" (UniqueName: \"kubernetes.io/projected/6d8b795e-518f-4834-b130-66f473ba4941-kube-api-access-x8sb6\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:49:53.062867 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:53.062864 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d8b795e-518f-4834-b130-66f473ba4941-maas-api-tls\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:49:53.211212 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:53.211125 2579 generic.go:356] "Generic (PLEG): container finished" podID="6d8b795e-518f-4834-b130-66f473ba4941" containerID="2f1578346ed09e0a599ada6679239b8e77dc3745f842936fff25e064ed90f511" exitCode=0 Jul 24 03:49:53.211212 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:53.211170 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x" Jul 24 03:49:53.211423 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:53.211209 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x" event={"ID":"6d8b795e-518f-4834-b130-66f473ba4941","Type":"ContainerDied","Data":"2f1578346ed09e0a599ada6679239b8e77dc3745f842936fff25e064ed90f511"} Jul 24 03:49:53.211423 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:53.211264 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x" event={"ID":"6d8b795e-518f-4834-b130-66f473ba4941","Type":"ContainerDied","Data":"42e22ca15466cae2d685086aa280b1099f14b0dd6e25c70072896320e3945cb9"} Jul 24 03:49:53.211423 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:53.211288 2579 scope.go:117] "RemoveContainer" containerID="2f1578346ed09e0a599ada6679239b8e77dc3745f842936fff25e064ed90f511" Jul 24 03:49:53.222143 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:53.222128 2579 scope.go:117] "RemoveContainer" containerID="2f1578346ed09e0a599ada6679239b8e77dc3745f842936fff25e064ed90f511" Jul 24 03:49:53.222412 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:49:53.222395 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f1578346ed09e0a599ada6679239b8e77dc3745f842936fff25e064ed90f511\": container with ID starting with 2f1578346ed09e0a599ada6679239b8e77dc3745f842936fff25e064ed90f511 not found: ID does not exist" containerID="2f1578346ed09e0a599ada6679239b8e77dc3745f842936fff25e064ed90f511" Jul 24 03:49:53.222478 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:53.222422 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f1578346ed09e0a599ada6679239b8e77dc3745f842936fff25e064ed90f511"} err="failed to get container status \"2f1578346ed09e0a599ada6679239b8e77dc3745f842936fff25e064ed90f511\": rpc error: code = NotFound desc = could not find container \"2f1578346ed09e0a599ada6679239b8e77dc3745f842936fff25e064ed90f511\": container with ID starting with 2f1578346ed09e0a599ada6679239b8e77dc3745f842936fff25e064ed90f511 not found: ID does not exist" Jul 24 03:49:53.230542 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:53.230516 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x"] Jul 24 03:49:53.234239 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:53.234220 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c4c112d8-97f6c5cff-rx94x"] Jul 24 03:49:55.033940 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:55.033908 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d8b795e-518f-4834-b130-66f473ba4941" path="/var/lib/kubelet/pods/6d8b795e-518f-4834-b130-66f473ba4941/volumes" Jul 24 03:49:57.274565 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:57.274526 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:49:58.468861 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:58.468819 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7"] Jul 24 03:49:58.469344 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:58.469052 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" podUID="4d52ad46-69f9-4167-9871-2530b387a632" containerName="istio-proxy" containerID="cri-o://9ecf3baebdd9a4b5a42f3c85c737e88d21ba3cd09fa20f6c39e75486a33acf15" gracePeriod=30 Jul 24 03:49:58.488203 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:58.488172 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:49:58.706836 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:58.706801 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:49:58.943972 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:58.943945 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:49:59.233439 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.233353 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9"] Jul 24 03:49:59.233999 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.233849 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d8b795e-518f-4834-b130-66f473ba4941" containerName="maas-api" Jul 24 03:49:59.233999 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.233873 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d8b795e-518f-4834-b130-66f473ba4941" containerName="maas-api" Jul 24 03:49:59.233999 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.233913 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e4d01cd-4ac8-46af-bcc9-57708025ff82" containerName="istio-proxy" Jul 24 03:49:59.233999 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.233923 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e4d01cd-4ac8-46af-bcc9-57708025ff82" containerName="istio-proxy" Jul 24 03:49:59.233999 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.233943 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e4d01cd-4ac8-46af-bcc9-57708025ff82" containerName="istio-proxy" Jul 24 03:49:59.233999 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.233952 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e4d01cd-4ac8-46af-bcc9-57708025ff82" containerName="istio-proxy" Jul 24 03:49:59.234396 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.234060 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e4d01cd-4ac8-46af-bcc9-57708025ff82" containerName="istio-proxy" Jul 24 03:49:59.234396 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.234074 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e4d01cd-4ac8-46af-bcc9-57708025ff82" containerName="istio-proxy" Jul 24 03:49:59.234396 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.234084 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d8b795e-518f-4834-b130-66f473ba4941" containerName="maas-api" Jul 24 03:49:59.238864 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.238843 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.242565 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.242547 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-0cd07409-openshift-default-dockercfg-f4knf\"" Jul 24 03:49:59.252547 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.252524 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9"] Jul 24 03:49:59.267661 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.267629 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:49:59.315999 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.315968 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8843e30b-1d95-4206-a139-9b92a566dd94-istio-token\") pod \"e2e-derive-0cd07409-openshift-default-696747c644-r9dx9\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.316140 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.316006 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nv4j\" (UniqueName: \"kubernetes.io/projected/8843e30b-1d95-4206-a139-9b92a566dd94-kube-api-access-9nv4j\") pod \"e2e-derive-0cd07409-openshift-default-696747c644-r9dx9\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.316140 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.316030 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-workload-certs\") pod \"e2e-derive-0cd07409-openshift-default-696747c644-r9dx9\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.316140 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.316057 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-workload-socket\") pod \"e2e-derive-0cd07409-openshift-default-696747c644-r9dx9\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.316140 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.316075 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-istio-data\") pod \"e2e-derive-0cd07409-openshift-default-696747c644-r9dx9\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.316140 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.316095 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8843e30b-1d95-4206-a139-9b92a566dd94-istio-podinfo\") pod \"e2e-derive-0cd07409-openshift-default-696747c644-r9dx9\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.316140 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.316110 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-credential-socket\") pod \"e2e-derive-0cd07409-openshift-default-696747c644-r9dx9\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.316140 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.316128 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8843e30b-1d95-4206-a139-9b92a566dd94-istiod-ca-cert\") pod \"e2e-derive-0cd07409-openshift-default-696747c644-r9dx9\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.316372 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.316187 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-istio-envoy\") pod \"e2e-derive-0cd07409-openshift-default-696747c644-r9dx9\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.417629 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.417590 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-workload-socket\") pod \"e2e-derive-0cd07409-openshift-default-696747c644-r9dx9\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.417629 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.417625 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-istio-data\") pod \"e2e-derive-0cd07409-openshift-default-696747c644-r9dx9\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.417886 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.417652 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8843e30b-1d95-4206-a139-9b92a566dd94-istio-podinfo\") pod \"e2e-derive-0cd07409-openshift-default-696747c644-r9dx9\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.417886 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.417674 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-credential-socket\") pod \"e2e-derive-0cd07409-openshift-default-696747c644-r9dx9\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.417886 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.417699 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8843e30b-1d95-4206-a139-9b92a566dd94-istiod-ca-cert\") pod \"e2e-derive-0cd07409-openshift-default-696747c644-r9dx9\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.417886 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.417743 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-istio-envoy\") pod \"e2e-derive-0cd07409-openshift-default-696747c644-r9dx9\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.417886 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.417863 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8843e30b-1d95-4206-a139-9b92a566dd94-istio-token\") pod \"e2e-derive-0cd07409-openshift-default-696747c644-r9dx9\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.418138 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.417886 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nv4j\" (UniqueName: \"kubernetes.io/projected/8843e30b-1d95-4206-a139-9b92a566dd94-kube-api-access-9nv4j\") pod \"e2e-derive-0cd07409-openshift-default-696747c644-r9dx9\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.418138 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.417921 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-workload-certs\") pod \"e2e-derive-0cd07409-openshift-default-696747c644-r9dx9\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.418138 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.418111 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-workload-socket\") pod \"e2e-derive-0cd07409-openshift-default-696747c644-r9dx9\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.418301 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.418251 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-workload-certs\") pod \"e2e-derive-0cd07409-openshift-default-696747c644-r9dx9\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.418391 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.418363 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-istio-data\") pod \"e2e-derive-0cd07409-openshift-default-696747c644-r9dx9\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.418453 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.418438 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-credential-socket\") pod \"e2e-derive-0cd07409-openshift-default-696747c644-r9dx9\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.418541 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.418451 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8843e30b-1d95-4206-a139-9b92a566dd94-istiod-ca-cert\") pod \"e2e-derive-0cd07409-openshift-default-696747c644-r9dx9\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.421058 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.421031 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-istio-envoy\") pod \"e2e-derive-0cd07409-openshift-default-696747c644-r9dx9\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.421213 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.421194 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8843e30b-1d95-4206-a139-9b92a566dd94-istio-podinfo\") pod \"e2e-derive-0cd07409-openshift-default-696747c644-r9dx9\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.426352 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.426333 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8843e30b-1d95-4206-a139-9b92a566dd94-istio-token\") pod \"e2e-derive-0cd07409-openshift-default-696747c644-r9dx9\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.426455 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.426438 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nv4j\" (UniqueName: \"kubernetes.io/projected/8843e30b-1d95-4206-a139-9b92a566dd94-kube-api-access-9nv4j\") pod \"e2e-derive-0cd07409-openshift-default-696747c644-r9dx9\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.460566 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.460540 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:49:59.550256 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.550180 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:49:59.691645 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.691616 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9"] Jul 24 03:49:59.693703 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:49:59.693676 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8843e30b_1d95_4206_a139_9b92a566dd94.slice/crio-8dd8430b448a44890b5217058de7fb5dbc4785f9d460b9a0c5a10353bff434c4 WatchSource:0}: Error finding container 8dd8430b448a44890b5217058de7fb5dbc4785f9d460b9a0c5a10353bff434c4: Status 404 returned error can't find the container with id 8dd8430b448a44890b5217058de7fb5dbc4785f9d460b9a0c5a10353bff434c4 Jul 24 03:49:59.696005 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.695974 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:49:59.696067 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.696037 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:49:59.696116 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:49:59.696070 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:50:00.251765 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:00.251728 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" event={"ID":"8843e30b-1d95-4206-a139-9b92a566dd94","Type":"ContainerStarted","Data":"0fefb2e4f052e161f6d57b55d0f8bf2b128c884fa63038667336a62d6442bd8a"} Jul 24 03:50:00.251987 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:00.251785 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" event={"ID":"8843e30b-1d95-4206-a139-9b92a566dd94","Type":"ContainerStarted","Data":"8dd8430b448a44890b5217058de7fb5dbc4785f9d460b9a0c5a10353bff434c4"} Jul 24 03:50:00.272584 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:00.272520 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" podStartSLOduration=1.27250163 podStartE2EDuration="1.27250163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:50:00.271122731 +0000 UTC m=+1949.710542192" watchObservedRunningTime="2026-07-24 03:50:00.27250163 +0000 UTC m=+1949.711921115" Jul 24 03:50:00.550526 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:00.550410 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:50:00.555558 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:00.555533 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:50:01.257495 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:01.257446 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:50:01.258608 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:01.258585 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:50:01.647056 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:01.647003 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:50:01.745156 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:01.745124 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:50:01.853628 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:01.853594 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq"] Jul 24 03:50:01.857744 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:01.857722 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq" Jul 24 03:50:01.862082 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:01.862060 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-0cd07409-serving-cert\"" Jul 24 03:50:01.876757 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:01.876732 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq"] Jul 24 03:50:01.919775 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:01.919676 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:50:01.942028 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:01.941981 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9ed72e-7308-44ea-8a1f-5238a4a13b97-maas-api-tls\") pod \"maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq\" (UID: \"ca9ed72e-7308-44ea-8a1f-5238a4a13b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq" Jul 24 03:50:01.942234 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:01.942077 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqpqz\" (UniqueName: \"kubernetes.io/projected/ca9ed72e-7308-44ea-8a1f-5238a4a13b97-kube-api-access-rqpqz\") pod \"maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq\" (UID: \"ca9ed72e-7308-44ea-8a1f-5238a4a13b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq" Jul 24 03:50:01.942234 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:01.942150 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9ed72e-7308-44ea-8a1f-5238a4a13b97-openshift-service-ca\") pod \"maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq\" (UID: \"ca9ed72e-7308-44ea-8a1f-5238a4a13b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq" Jul 24 03:50:02.024770 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:02.024731 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:50:02.043132 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:02.043108 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqpqz\" (UniqueName: \"kubernetes.io/projected/ca9ed72e-7308-44ea-8a1f-5238a4a13b97-kube-api-access-rqpqz\") pod \"maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq\" (UID: \"ca9ed72e-7308-44ea-8a1f-5238a4a13b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq" Jul 24 03:50:02.043286 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:02.043155 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9ed72e-7308-44ea-8a1f-5238a4a13b97-openshift-service-ca\") pod \"maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq\" (UID: \"ca9ed72e-7308-44ea-8a1f-5238a4a13b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq" Jul 24 03:50:02.043286 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:02.043216 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9ed72e-7308-44ea-8a1f-5238a4a13b97-maas-api-tls\") pod \"maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq\" (UID: \"ca9ed72e-7308-44ea-8a1f-5238a4a13b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq" Jul 24 03:50:02.043896 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:02.043878 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9ed72e-7308-44ea-8a1f-5238a4a13b97-openshift-service-ca\") pod \"maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq\" (UID: \"ca9ed72e-7308-44ea-8a1f-5238a4a13b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq" Jul 24 03:50:02.045742 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:02.045722 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9ed72e-7308-44ea-8a1f-5238a4a13b97-maas-api-tls\") pod \"maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq\" (UID: \"ca9ed72e-7308-44ea-8a1f-5238a4a13b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq" Jul 24 03:50:02.057792 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:02.057769 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqpqz\" (UniqueName: \"kubernetes.io/projected/ca9ed72e-7308-44ea-8a1f-5238a4a13b97-kube-api-access-rqpqz\") pod \"maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq\" (UID: \"ca9ed72e-7308-44ea-8a1f-5238a4a13b97\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq" Jul 24 03:50:02.170195 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:02.170107 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq" Jul 24 03:50:02.313020 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:02.312995 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq"] Jul 24 03:50:02.315069 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:50:02.315043 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9ed72e_7308_44ea_8a1f_5238a4a13b97.slice/crio-9752bd8bfd0ab4fc446ff7741efbbdb1213a0f5225f59c4b03233aec79a7df12 WatchSource:0}: Error finding container 9752bd8bfd0ab4fc446ff7741efbbdb1213a0f5225f59c4b03233aec79a7df12: Status 404 returned error can't find the container with id 9752bd8bfd0ab4fc446ff7741efbbdb1213a0f5225f59c4b03233aec79a7df12 Jul 24 03:50:02.316929 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:02.316912 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 03:50:02.428526 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:02.428410 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:50:02.536934 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:02.536893 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:50:03.272191 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.272156 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq" event={"ID":"ca9ed72e-7308-44ea-8a1f-5238a4a13b97","Type":"ContainerStarted","Data":"525ade62ecfe89a0ecf70cf9851f98b6d1ed71d9498270e9b3f63407e463779b"} Jul 24 03:50:03.272700 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.272224 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq" event={"ID":"ca9ed72e-7308-44ea-8a1f-5238a4a13b97","Type":"ContainerStarted","Data":"9752bd8bfd0ab4fc446ff7741efbbdb1213a0f5225f59c4b03233aec79a7df12"} Jul 24 03:50:03.289908 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.289838 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq" podStartSLOduration=1.801986082 podStartE2EDuration="2.289818923s" totalImagesPullingTime="487.832841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:50:03.289082481 +0000 UTC m=+1952.728501941" watchObservedRunningTime="2026-07-24 03:50:03.289818923 +0000 UTC m=+1952.729238384" Jul 24 03:50:03.732173 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.732149 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:50:03.861393 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.861363 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-istio-data\") pod \"4d52ad46-69f9-4167-9871-2530b387a632\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " Jul 24 03:50:03.861643 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.861404 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-credential-socket\") pod \"4d52ad46-69f9-4167-9871-2530b387a632\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " Jul 24 03:50:03.861643 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.861427 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-workload-certs\") pod \"4d52ad46-69f9-4167-9871-2530b387a632\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " Jul 24 03:50:03.861643 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.861449 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-istio-envoy\") pod \"4d52ad46-69f9-4167-9871-2530b387a632\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " Jul 24 03:50:03.861643 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.861522 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d52ad46-69f9-4167-9871-2530b387a632-istio-token\") pod \"4d52ad46-69f9-4167-9871-2530b387a632\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " Jul 24 03:50:03.861643 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.861545 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d52ad46-69f9-4167-9871-2530b387a632-istio-podinfo\") pod \"4d52ad46-69f9-4167-9871-2530b387a632\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " Jul 24 03:50:03.861643 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.861576 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d52ad46-69f9-4167-9871-2530b387a632-istiod-ca-cert\") pod \"4d52ad46-69f9-4167-9871-2530b387a632\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " Jul 24 03:50:03.861643 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.861610 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gcch\" (UniqueName: \"kubernetes.io/projected/4d52ad46-69f9-4167-9871-2530b387a632-kube-api-access-2gcch\") pod \"4d52ad46-69f9-4167-9871-2530b387a632\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " Jul 24 03:50:03.861993 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.861708 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-workload-socket\") pod \"4d52ad46-69f9-4167-9871-2530b387a632\" (UID: \"4d52ad46-69f9-4167-9871-2530b387a632\") " Jul 24 03:50:03.861993 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.861711 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4d52ad46-69f9-4167-9871-2530b387a632" (UID: "4d52ad46-69f9-4167-9871-2530b387a632"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:50:03.862112 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.862029 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-credential-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:50:03.862112 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.862023 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d52ad46-69f9-4167-9871-2530b387a632-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4d52ad46-69f9-4167-9871-2530b387a632" (UID: "4d52ad46-69f9-4167-9871-2530b387a632"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 03:50:03.862112 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.862085 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4d52ad46-69f9-4167-9871-2530b387a632" (UID: "4d52ad46-69f9-4167-9871-2530b387a632"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:50:03.862260 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.862243 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4d52ad46-69f9-4167-9871-2530b387a632" (UID: "4d52ad46-69f9-4167-9871-2530b387a632"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:50:03.862562 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.862531 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-istio-data" (OuterVolumeSpecName: "istio-data") pod "4d52ad46-69f9-4167-9871-2530b387a632" (UID: "4d52ad46-69f9-4167-9871-2530b387a632"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:50:03.863962 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.863938 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4d52ad46-69f9-4167-9871-2530b387a632" (UID: "4d52ad46-69f9-4167-9871-2530b387a632"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:50:03.864050 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.864005 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d52ad46-69f9-4167-9871-2530b387a632-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4d52ad46-69f9-4167-9871-2530b387a632" (UID: "4d52ad46-69f9-4167-9871-2530b387a632"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 03:50:03.864050 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.864022 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d52ad46-69f9-4167-9871-2530b387a632-kube-api-access-2gcch" (OuterVolumeSpecName: "kube-api-access-2gcch") pod "4d52ad46-69f9-4167-9871-2530b387a632" (UID: "4d52ad46-69f9-4167-9871-2530b387a632"). InnerVolumeSpecName "kube-api-access-2gcch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:50:03.864124 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.864072 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d52ad46-69f9-4167-9871-2530b387a632-istio-token" (OuterVolumeSpecName: "istio-token") pod "4d52ad46-69f9-4167-9871-2530b387a632" (UID: "4d52ad46-69f9-4167-9871-2530b387a632"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:50:03.962953 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.962913 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-workload-certs\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:50:03.962953 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.962953 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-istio-envoy\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:50:03.962953 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.962963 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d52ad46-69f9-4167-9871-2530b387a632-istio-token\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:50:03.963182 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.962971 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d52ad46-69f9-4167-9871-2530b387a632-istio-podinfo\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:50:03.963182 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.962979 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d52ad46-69f9-4167-9871-2530b387a632-istiod-ca-cert\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:50:03.963182 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.962988 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gcch\" (UniqueName: \"kubernetes.io/projected/4d52ad46-69f9-4167-9871-2530b387a632-kube-api-access-2gcch\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:50:03.963182 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.962997 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-workload-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:50:03.963182 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:03.963005 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d52ad46-69f9-4167-9871-2530b387a632-istio-data\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:50:04.279123 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:04.279030 2579 generic.go:356] "Generic (PLEG): container finished" podID="4d52ad46-69f9-4167-9871-2530b387a632" containerID="9ecf3baebdd9a4b5a42f3c85c737e88d21ba3cd09fa20f6c39e75486a33acf15" exitCode=0 Jul 24 03:50:04.279123 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:04.279068 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" event={"ID":"4d52ad46-69f9-4167-9871-2530b387a632","Type":"ContainerDied","Data":"9ecf3baebdd9a4b5a42f3c85c737e88d21ba3cd09fa20f6c39e75486a33acf15"} Jul 24 03:50:04.279123 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:04.279082 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" Jul 24 03:50:04.279123 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:04.279123 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7" event={"ID":"4d52ad46-69f9-4167-9871-2530b387a632","Type":"ContainerDied","Data":"8df71dfbf8b966d954338586b75d15f8f06f41e70ab5bf8d18dae9c0166f4684"} Jul 24 03:50:04.279756 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:04.279154 2579 scope.go:117] "RemoveContainer" containerID="9ecf3baebdd9a4b5a42f3c85c737e88d21ba3cd09fa20f6c39e75486a33acf15" Jul 24 03:50:04.279756 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:04.279508 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq" Jul 24 03:50:04.291131 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:04.291108 2579 scope.go:117] "RemoveContainer" containerID="ede3abcd4d2adf105fd7d92d97ae74927e0cc291ef30a9f26e7bbbb5c1327cd2" Jul 24 03:50:04.300335 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:04.300317 2579 scope.go:117] "RemoveContainer" containerID="9ecf3baebdd9a4b5a42f3c85c737e88d21ba3cd09fa20f6c39e75486a33acf15" Jul 24 03:50:04.300669 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:50:04.300650 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ecf3baebdd9a4b5a42f3c85c737e88d21ba3cd09fa20f6c39e75486a33acf15\": container with ID starting with 9ecf3baebdd9a4b5a42f3c85c737e88d21ba3cd09fa20f6c39e75486a33acf15 not found: ID does not exist" containerID="9ecf3baebdd9a4b5a42f3c85c737e88d21ba3cd09fa20f6c39e75486a33acf15" Jul 24 03:50:04.300740 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:04.300683 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ecf3baebdd9a4b5a42f3c85c737e88d21ba3cd09fa20f6c39e75486a33acf15"} err="failed to get container status \"9ecf3baebdd9a4b5a42f3c85c737e88d21ba3cd09fa20f6c39e75486a33acf15\": rpc error: code = NotFound desc = could not find container \"9ecf3baebdd9a4b5a42f3c85c737e88d21ba3cd09fa20f6c39e75486a33acf15\": container with ID starting with 9ecf3baebdd9a4b5a42f3c85c737e88d21ba3cd09fa20f6c39e75486a33acf15 not found: ID does not exist" Jul 24 03:50:04.300740 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:04.300700 2579 scope.go:117] "RemoveContainer" containerID="ede3abcd4d2adf105fd7d92d97ae74927e0cc291ef30a9f26e7bbbb5c1327cd2" Jul 24 03:50:04.300945 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:50:04.300930 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede3abcd4d2adf105fd7d92d97ae74927e0cc291ef30a9f26e7bbbb5c1327cd2\": container with ID starting with ede3abcd4d2adf105fd7d92d97ae74927e0cc291ef30a9f26e7bbbb5c1327cd2 not found: ID does not exist" containerID="ede3abcd4d2adf105fd7d92d97ae74927e0cc291ef30a9f26e7bbbb5c1327cd2" Jul 24 03:50:04.300985 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:04.300949 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede3abcd4d2adf105fd7d92d97ae74927e0cc291ef30a9f26e7bbbb5c1327cd2"} err="failed to get container status \"ede3abcd4d2adf105fd7d92d97ae74927e0cc291ef30a9f26e7bbbb5c1327cd2\": rpc error: code = NotFound desc = could not find container \"ede3abcd4d2adf105fd7d92d97ae74927e0cc291ef30a9f26e7bbbb5c1327cd2\": container with ID starting with ede3abcd4d2adf105fd7d92d97ae74927e0cc291ef30a9f26e7bbbb5c1327cd2 not found: ID does not exist" Jul 24 03:50:04.301696 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:04.301677 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7"] Jul 24 03:50:04.306219 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:04.306193 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c4c112d8-openshift-default-6ff765989b-ksft7"] Jul 24 03:50:05.034260 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:05.034230 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d52ad46-69f9-4167-9871-2530b387a632" path="/var/lib/kubelet/pods/4d52ad46-69f9-4167-9871-2530b387a632/volumes" Jul 24 03:50:10.297943 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:10.297911 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq" Jul 24 03:50:15.341739 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:15.341705 2579 generic.go:356] "Generic (PLEG): container finished" podID="8843e30b-1d95-4206-a139-9b92a566dd94" containerID="0fefb2e4f052e161f6d57b55d0f8bf2b128c884fa63038667336a62d6442bd8a" exitCode=0 Jul 24 03:50:15.342213 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:15.341765 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" event={"ID":"8843e30b-1d95-4206-a139-9b92a566dd94","Type":"ContainerDied","Data":"0fefb2e4f052e161f6d57b55d0f8bf2b128c884fa63038667336a62d6442bd8a"} Jul 24 03:50:15.342213 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:15.342199 2579 scope.go:117] "RemoveContainer" containerID="0fefb2e4f052e161f6d57b55d0f8bf2b128c884fa63038667336a62d6442bd8a" Jul 24 03:50:15.343393 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:15.343360 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:50:15.343526 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:15.343446 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:50:15.343602 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:15.343526 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:50:16.257899 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:16.257871 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:50:16.349636 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:16.349601 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" event={"ID":"8843e30b-1d95-4206-a139-9b92a566dd94","Type":"ContainerStarted","Data":"5dc2717c57895f500406e07c1a9648cf0af8f32df5f16a016c5947a49f854981"} Jul 24 03:50:16.551015 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:16.550922 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:50:16.552069 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:16.552047 2579 patch_prober.go:28] interesting pod/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 24 03:50:16.552115 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:16.552098 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" podUID="8843e30b-1d95-4206-a139-9b92a566dd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 24 03:50:17.551102 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:17.551062 2579 patch_prober.go:28] interesting pod/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 24 03:50:17.551592 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:17.551124 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" podUID="8843e30b-1d95-4206-a139-9b92a566dd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 24 03:50:18.550694 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:18.550660 2579 patch_prober.go:28] interesting pod/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 24 03:50:18.550873 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:18.550712 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" podUID="8843e30b-1d95-4206-a139-9b92a566dd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 24 03:50:19.550888 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:19.550853 2579 patch_prober.go:28] interesting pod/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 24 03:50:19.551284 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:19.550904 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" podUID="8843e30b-1d95-4206-a139-9b92a566dd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 24 03:50:20.554769 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:20.554739 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:50:20.555183 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:20.555036 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:50:20.555628 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:20.555609 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:50:37.002730 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:37.002684 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq"] Jul 24 03:50:37.003551 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:37.003479 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq" podUID="ca9ed72e-7308-44ea-8a1f-5238a4a13b97" containerName="maas-api" containerID="cri-o://525ade62ecfe89a0ecf70cf9851f98b6d1ed71d9498270e9b3f63407e463779b" gracePeriod=30 Jul 24 03:50:37.030662 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:37.030620 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:50:37.101260 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:37.101225 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:50:37.165167 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:37.165129 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:50:37.266904 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:37.266838 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq" Jul 24 03:50:37.391505 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:37.391454 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqpqz\" (UniqueName: \"kubernetes.io/projected/ca9ed72e-7308-44ea-8a1f-5238a4a13b97-kube-api-access-rqpqz\") pod \"ca9ed72e-7308-44ea-8a1f-5238a4a13b97\" (UID: \"ca9ed72e-7308-44ea-8a1f-5238a4a13b97\") " Jul 24 03:50:37.391668 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:37.391573 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9ed72e-7308-44ea-8a1f-5238a4a13b97-openshift-service-ca\") pod \"ca9ed72e-7308-44ea-8a1f-5238a4a13b97\" (UID: \"ca9ed72e-7308-44ea-8a1f-5238a4a13b97\") " Jul 24 03:50:37.391668 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:37.391621 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9ed72e-7308-44ea-8a1f-5238a4a13b97-maas-api-tls\") pod \"ca9ed72e-7308-44ea-8a1f-5238a4a13b97\" (UID: \"ca9ed72e-7308-44ea-8a1f-5238a4a13b97\") " Jul 24 03:50:37.391959 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:37.391938 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca9ed72e-7308-44ea-8a1f-5238a4a13b97-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ca9ed72e-7308-44ea-8a1f-5238a4a13b97" (UID: "ca9ed72e-7308-44ea-8a1f-5238a4a13b97"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 03:50:37.393790 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:37.393763 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca9ed72e-7308-44ea-8a1f-5238a4a13b97-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ca9ed72e-7308-44ea-8a1f-5238a4a13b97" (UID: "ca9ed72e-7308-44ea-8a1f-5238a4a13b97"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 03:50:37.393790 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:37.393776 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca9ed72e-7308-44ea-8a1f-5238a4a13b97-kube-api-access-rqpqz" (OuterVolumeSpecName: "kube-api-access-rqpqz") pod "ca9ed72e-7308-44ea-8a1f-5238a4a13b97" (UID: "ca9ed72e-7308-44ea-8a1f-5238a4a13b97"). InnerVolumeSpecName "kube-api-access-rqpqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:50:37.464217 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:37.464178 2579 generic.go:356] "Generic (PLEG): container finished" podID="ca9ed72e-7308-44ea-8a1f-5238a4a13b97" containerID="525ade62ecfe89a0ecf70cf9851f98b6d1ed71d9498270e9b3f63407e463779b" exitCode=0 Jul 24 03:50:37.464391 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:37.464226 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq" Jul 24 03:50:37.464391 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:37.464259 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq" event={"ID":"ca9ed72e-7308-44ea-8a1f-5238a4a13b97","Type":"ContainerDied","Data":"525ade62ecfe89a0ecf70cf9851f98b6d1ed71d9498270e9b3f63407e463779b"} Jul 24 03:50:37.464391 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:37.464333 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq" event={"ID":"ca9ed72e-7308-44ea-8a1f-5238a4a13b97","Type":"ContainerDied","Data":"9752bd8bfd0ab4fc446ff7741efbbdb1213a0f5225f59c4b03233aec79a7df12"} Jul 24 03:50:37.464391 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:37.464371 2579 scope.go:117] "RemoveContainer" containerID="525ade62ecfe89a0ecf70cf9851f98b6d1ed71d9498270e9b3f63407e463779b" Jul 24 03:50:37.475041 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:37.475016 2579 scope.go:117] "RemoveContainer" containerID="525ade62ecfe89a0ecf70cf9851f98b6d1ed71d9498270e9b3f63407e463779b" Jul 24 03:50:37.475364 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:50:37.475336 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"525ade62ecfe89a0ecf70cf9851f98b6d1ed71d9498270e9b3f63407e463779b\": container with ID starting with 525ade62ecfe89a0ecf70cf9851f98b6d1ed71d9498270e9b3f63407e463779b not found: ID does not exist" containerID="525ade62ecfe89a0ecf70cf9851f98b6d1ed71d9498270e9b3f63407e463779b" Jul 24 03:50:37.475478 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:37.475375 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"525ade62ecfe89a0ecf70cf9851f98b6d1ed71d9498270e9b3f63407e463779b"} err="failed to get container status \"525ade62ecfe89a0ecf70cf9851f98b6d1ed71d9498270e9b3f63407e463779b\": rpc error: code = NotFound desc = could not find container \"525ade62ecfe89a0ecf70cf9851f98b6d1ed71d9498270e9b3f63407e463779b\": container with ID starting with 525ade62ecfe89a0ecf70cf9851f98b6d1ed71d9498270e9b3f63407e463779b not found: ID does not exist" Jul 24 03:50:37.487064 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:37.487037 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq"] Jul 24 03:50:37.490971 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:37.490944 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0cd07409-86ddd7dcb5-p9vfq"] Jul 24 03:50:37.492548 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:37.492528 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqpqz\" (UniqueName: \"kubernetes.io/projected/ca9ed72e-7308-44ea-8a1f-5238a4a13b97-kube-api-access-rqpqz\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:50:37.492634 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:37.492553 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9ed72e-7308-44ea-8a1f-5238a4a13b97-openshift-service-ca\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:50:37.492634 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:37.492570 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9ed72e-7308-44ea-8a1f-5238a4a13b97-maas-api-tls\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:50:37.497908 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:50:37.497886 2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9ed72e_7308_44ea_8a1f_5238a4a13b97.slice/crio-9752bd8bfd0ab4fc446ff7741efbbdb1213a0f5225f59c4b03233aec79a7df12\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9ed72e_7308_44ea_8a1f_5238a4a13b97.slice\": RecentStats: unable to find data in memory cache]" Jul 24 03:50:39.034062 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:39.034028 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca9ed72e-7308-44ea-8a1f-5238a4a13b97" path="/var/lib/kubelet/pods/ca9ed72e-7308-44ea-8a1f-5238a4a13b97/volumes" Jul 24 03:50:41.556920 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:41.556884 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:50:41.691487 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:41.691428 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9"] Jul 24 03:50:41.691707 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:41.691680 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" podUID="8843e30b-1d95-4206-a139-9b92a566dd94" containerName="istio-proxy" containerID="cri-o://5dc2717c57895f500406e07c1a9648cf0af8f32df5f16a016c5947a49f854981" gracePeriod=30 Jul 24 03:50:41.711941 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:41.711915 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:50:41.963069 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:41.963034 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:50:42.447658 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:42.447611 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:50:42.552828 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:42.552788 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:50:42.728034 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:42.727946 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:50:44.130656 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:44.130629 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:50:46.945044 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:46.945020 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:50:47.084156 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.084078 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8843e30b-1d95-4206-a139-9b92a566dd94-istiod-ca-cert\") pod \"8843e30b-1d95-4206-a139-9b92a566dd94\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " Jul 24 03:50:47.084320 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.084160 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-workload-socket\") pod \"8843e30b-1d95-4206-a139-9b92a566dd94\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " Jul 24 03:50:47.084320 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.084186 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8843e30b-1d95-4206-a139-9b92a566dd94-istio-token\") pod \"8843e30b-1d95-4206-a139-9b92a566dd94\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " Jul 24 03:50:47.084320 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.084224 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nv4j\" (UniqueName: \"kubernetes.io/projected/8843e30b-1d95-4206-a139-9b92a566dd94-kube-api-access-9nv4j\") pod \"8843e30b-1d95-4206-a139-9b92a566dd94\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " Jul 24 03:50:47.084320 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.084249 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-credential-socket\") pod \"8843e30b-1d95-4206-a139-9b92a566dd94\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " Jul 24 03:50:47.084320 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.084276 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-istio-envoy\") pod \"8843e30b-1d95-4206-a139-9b92a566dd94\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " Jul 24 03:50:47.084320 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.084310 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-workload-certs\") pod \"8843e30b-1d95-4206-a139-9b92a566dd94\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " Jul 24 03:50:47.084646 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.084365 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-istio-data\") pod \"8843e30b-1d95-4206-a139-9b92a566dd94\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " Jul 24 03:50:47.084646 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.084386 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8843e30b-1d95-4206-a139-9b92a566dd94-istio-podinfo\") pod \"8843e30b-1d95-4206-a139-9b92a566dd94\" (UID: \"8843e30b-1d95-4206-a139-9b92a566dd94\") " Jul 24 03:50:47.084646 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.084507 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8843e30b-1d95-4206-a139-9b92a566dd94" (UID: "8843e30b-1d95-4206-a139-9b92a566dd94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:50:47.084646 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.084536 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8843e30b-1d95-4206-a139-9b92a566dd94-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8843e30b-1d95-4206-a139-9b92a566dd94" (UID: "8843e30b-1d95-4206-a139-9b92a566dd94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 03:50:47.084867 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.084646 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8843e30b-1d95-4206-a139-9b92a566dd94" (UID: "8843e30b-1d95-4206-a139-9b92a566dd94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:50:47.084867 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.084672 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8843e30b-1d95-4206-a139-9b92a566dd94" (UID: "8843e30b-1d95-4206-a139-9b92a566dd94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:50:47.084969 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.084875 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-workload-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:50:47.084969 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.084897 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-credential-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:50:47.084969 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.084911 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-workload-certs\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:50:47.084969 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.084925 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8843e30b-1d95-4206-a139-9b92a566dd94-istiod-ca-cert\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:50:47.085353 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.085333 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-istio-data" (OuterVolumeSpecName: "istio-data") pod "8843e30b-1d95-4206-a139-9b92a566dd94" (UID: "8843e30b-1d95-4206-a139-9b92a566dd94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:50:47.086687 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.086656 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8843e30b-1d95-4206-a139-9b92a566dd94-kube-api-access-9nv4j" (OuterVolumeSpecName: "kube-api-access-9nv4j") pod "8843e30b-1d95-4206-a139-9b92a566dd94" (UID: "8843e30b-1d95-4206-a139-9b92a566dd94"). InnerVolumeSpecName "kube-api-access-9nv4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:50:47.086824 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.086758 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8843e30b-1d95-4206-a139-9b92a566dd94-istio-token" (OuterVolumeSpecName: "istio-token") pod "8843e30b-1d95-4206-a139-9b92a566dd94" (UID: "8843e30b-1d95-4206-a139-9b92a566dd94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:50:47.087206 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.087180 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8843e30b-1d95-4206-a139-9b92a566dd94-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8843e30b-1d95-4206-a139-9b92a566dd94" (UID: "8843e30b-1d95-4206-a139-9b92a566dd94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 03:50:47.087206 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.087180 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8843e30b-1d95-4206-a139-9b92a566dd94" (UID: "8843e30b-1d95-4206-a139-9b92a566dd94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:50:47.185541 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.185511 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-istio-envoy\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:50:47.185541 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.185535 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8843e30b-1d95-4206-a139-9b92a566dd94-istio-data\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:50:47.185541 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.185544 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8843e30b-1d95-4206-a139-9b92a566dd94-istio-podinfo\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:50:47.185763 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.185552 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8843e30b-1d95-4206-a139-9b92a566dd94-istio-token\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:50:47.185763 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.185561 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nv4j\" (UniqueName: \"kubernetes.io/projected/8843e30b-1d95-4206-a139-9b92a566dd94-kube-api-access-9nv4j\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:50:47.522080 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.522041 2579 generic.go:356] "Generic (PLEG): container finished" podID="8843e30b-1d95-4206-a139-9b92a566dd94" containerID="5dc2717c57895f500406e07c1a9648cf0af8f32df5f16a016c5947a49f854981" exitCode=0 Jul 24 03:50:47.522300 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.522086 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" Jul 24 03:50:47.522300 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.522111 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" event={"ID":"8843e30b-1d95-4206-a139-9b92a566dd94","Type":"ContainerDied","Data":"5dc2717c57895f500406e07c1a9648cf0af8f32df5f16a016c5947a49f854981"} Jul 24 03:50:47.522300 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.522175 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9" event={"ID":"8843e30b-1d95-4206-a139-9b92a566dd94","Type":"ContainerDied","Data":"8dd8430b448a44890b5217058de7fb5dbc4785f9d460b9a0c5a10353bff434c4"} Jul 24 03:50:47.522300 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.522208 2579 scope.go:117] "RemoveContainer" containerID="5dc2717c57895f500406e07c1a9648cf0af8f32df5f16a016c5947a49f854981" Jul 24 03:50:47.532857 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.532838 2579 scope.go:117] "RemoveContainer" containerID="0fefb2e4f052e161f6d57b55d0f8bf2b128c884fa63038667336a62d6442bd8a" Jul 24 03:50:47.541320 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.541297 2579 scope.go:117] "RemoveContainer" containerID="5dc2717c57895f500406e07c1a9648cf0af8f32df5f16a016c5947a49f854981" Jul 24 03:50:47.541698 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:50:47.541668 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dc2717c57895f500406e07c1a9648cf0af8f32df5f16a016c5947a49f854981\": container with ID starting with 5dc2717c57895f500406e07c1a9648cf0af8f32df5f16a016c5947a49f854981 not found: ID does not exist" containerID="5dc2717c57895f500406e07c1a9648cf0af8f32df5f16a016c5947a49f854981" Jul 24 03:50:47.541776 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.541709 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dc2717c57895f500406e07c1a9648cf0af8f32df5f16a016c5947a49f854981"} err="failed to get container status \"5dc2717c57895f500406e07c1a9648cf0af8f32df5f16a016c5947a49f854981\": rpc error: code = NotFound desc = could not find container \"5dc2717c57895f500406e07c1a9648cf0af8f32df5f16a016c5947a49f854981\": container with ID starting with 5dc2717c57895f500406e07c1a9648cf0af8f32df5f16a016c5947a49f854981 not found: ID does not exist" Jul 24 03:50:47.541776 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.541732 2579 scope.go:117] "RemoveContainer" containerID="0fefb2e4f052e161f6d57b55d0f8bf2b128c884fa63038667336a62d6442bd8a" Jul 24 03:50:47.542017 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:50:47.542002 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fefb2e4f052e161f6d57b55d0f8bf2b128c884fa63038667336a62d6442bd8a\": container with ID starting with 0fefb2e4f052e161f6d57b55d0f8bf2b128c884fa63038667336a62d6442bd8a not found: ID does not exist" containerID="0fefb2e4f052e161f6d57b55d0f8bf2b128c884fa63038667336a62d6442bd8a" Jul 24 03:50:47.542080 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.542020 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fefb2e4f052e161f6d57b55d0f8bf2b128c884fa63038667336a62d6442bd8a"} err="failed to get container status \"0fefb2e4f052e161f6d57b55d0f8bf2b128c884fa63038667336a62d6442bd8a\": rpc error: code = NotFound desc = could not find container \"0fefb2e4f052e161f6d57b55d0f8bf2b128c884fa63038667336a62d6442bd8a\": container with ID starting with 0fefb2e4f052e161f6d57b55d0f8bf2b128c884fa63038667336a62d6442bd8a not found: ID does not exist" Jul 24 03:50:47.550319 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.550296 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9"] Jul 24 03:50:47.558136 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:47.558111 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-0cd07409-openshift-default-696747c644-r9dx9"] Jul 24 03:50:49.034861 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:49.034827 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8843e30b-1d95-4206-a139-9b92a566dd94" path="/var/lib/kubelet/pods/8843e30b-1d95-4206-a139-9b92a566dd94/volumes" Jul 24 03:50:49.731167 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:49.731121 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:50:49.835314 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:49.835267 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:50:49.932680 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:49.932651 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:50:50.032818 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:50.032714 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:50:50.465142 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:50:50.465108 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:51:16.489537 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:51:16.489502 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:51:22.793331 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:51:22.793284 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:51:22.865147 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:51:22.865110 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:51:22.959679 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:51:22.959636 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:51:39.867681 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:51:39.867622 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:51:45.525091 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:51:45.525055 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:51:45.644419 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:51:45.644386 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:51:45.757396 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:51:45.757360 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:51:56.271234 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:51:56.271162 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:51:57.060662 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:51:57.060628 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:51:57.436533 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:51:57.436445 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:51:57.492419 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:51:57.492388 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:03.628887 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:03.628854 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:03.733388 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:03.733348 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:03.829164 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:03.829132 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:03.929007 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:03.928916 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:04.693896 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:04.693819 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:04.789823 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:04.789794 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:04.920337 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:04.920309 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:06.503299 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:06.503264 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:12.675889 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:12.675852 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:12.779965 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:12.779927 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:13.398368 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:13.398325 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:19.312156 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.312109 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:19.410985 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.410932 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:19.546657 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.546627 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:19.841379 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.841340 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln"] Jul 24 03:52:19.841757 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.841743 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8843e30b-1d95-4206-a139-9b92a566dd94" containerName="istio-proxy" Jul 24 03:52:19.841757 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.841758 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8843e30b-1d95-4206-a139-9b92a566dd94" containerName="istio-proxy" Jul 24 03:52:19.841873 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.841772 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca9ed72e-7308-44ea-8a1f-5238a4a13b97" containerName="maas-api" Jul 24 03:52:19.841873 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.841778 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca9ed72e-7308-44ea-8a1f-5238a4a13b97" containerName="maas-api" Jul 24 03:52:19.841873 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.841795 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d52ad46-69f9-4167-9871-2530b387a632" containerName="istio-proxy" Jul 24 03:52:19.841873 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.841800 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d52ad46-69f9-4167-9871-2530b387a632" containerName="istio-proxy" Jul 24 03:52:19.841873 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.841809 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d52ad46-69f9-4167-9871-2530b387a632" containerName="istio-proxy" Jul 24 03:52:19.841873 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.841814 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d52ad46-69f9-4167-9871-2530b387a632" containerName="istio-proxy" Jul 24 03:52:19.841873 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.841873 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8843e30b-1d95-4206-a139-9b92a566dd94" containerName="istio-proxy" Jul 24 03:52:19.842224 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.841881 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8843e30b-1d95-4206-a139-9b92a566dd94" containerName="istio-proxy" Jul 24 03:52:19.842224 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.841891 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca9ed72e-7308-44ea-8a1f-5238a4a13b97" containerName="maas-api" Jul 24 03:52:19.842224 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.841898 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d52ad46-69f9-4167-9871-2530b387a632" containerName="istio-proxy" Jul 24 03:52:19.842224 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.841904 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d52ad46-69f9-4167-9871-2530b387a632" containerName="istio-proxy" Jul 24 03:52:19.849797 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.849769 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:19.852628 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.852603 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-365be7db-openshift-default-dockercfg-4mjct\"" Jul 24 03:52:19.861046 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.861021 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:19.865596 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.865564 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln"] Jul 24 03:52:19.958514 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.958454 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-istio-data\") pod \"e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:19.958703 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.958568 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-workload-certs\") pod \"e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:19.958703 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.958631 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-workload-socket\") pod \"e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:19.958703 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.958651 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ce924a5-8762-48cd-a0de-29816cf6a755-istio-podinfo\") pod \"e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:19.958703 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.958672 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-credential-socket\") pod \"e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:19.958703 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.958693 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m9hw\" (UniqueName: \"kubernetes.io/projected/0ce924a5-8762-48cd-a0de-29816cf6a755-kube-api-access-2m9hw\") pod \"e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:19.959230 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.958811 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ce924a5-8762-48cd-a0de-29816cf6a755-istio-token\") pod \"e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:19.959230 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.958848 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ce924a5-8762-48cd-a0de-29816cf6a755-istiod-ca-cert\") pod \"e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:19.959230 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:19.958884 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-istio-envoy\") pod \"e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:20.055448 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:20.055403 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:20.059663 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:20.059629 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ce924a5-8762-48cd-a0de-29816cf6a755-istiod-ca-cert\") pod \"e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:20.059796 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:20.059676 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-istio-envoy\") pod \"e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:20.059796 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:20.059732 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-istio-data\") pod \"e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:20.059796 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:20.059779 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-workload-certs\") pod \"e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:20.059965 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:20.059819 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-workload-socket\") pod \"e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:20.059965 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:20.059846 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ce924a5-8762-48cd-a0de-29816cf6a755-istio-podinfo\") pod \"e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:20.059965 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:20.059877 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-credential-socket\") pod \"e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:20.059965 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:20.059914 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m9hw\" (UniqueName: \"kubernetes.io/projected/0ce924a5-8762-48cd-a0de-29816cf6a755-kube-api-access-2m9hw\") pod \"e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:20.060160 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:20.059985 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ce924a5-8762-48cd-a0de-29816cf6a755-istio-token\") pod \"e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:20.060234 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:20.060209 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-istio-data\") pod \"e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:20.060352 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:20.060331 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-workload-certs\") pod \"e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:20.060424 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:20.060404 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ce924a5-8762-48cd-a0de-29816cf6a755-istiod-ca-cert\") pod \"e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:20.060535 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:20.060512 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-workload-socket\") pod \"e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:20.060655 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:20.060437 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-credential-socket\") pod \"e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:20.062122 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:20.062103 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-istio-envoy\") pod \"e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:20.062407 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:20.062388 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ce924a5-8762-48cd-a0de-29816cf6a755-istio-podinfo\") pod \"e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:20.068306 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:20.068284 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m9hw\" (UniqueName: \"kubernetes.io/projected/0ce924a5-8762-48cd-a0de-29816cf6a755-kube-api-access-2m9hw\") pod \"e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:20.068405 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:20.068387 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ce924a5-8762-48cd-a0de-29816cf6a755-istio-token\") pod \"e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:20.161154 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:20.161119 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:20.301183 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:20.301157 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln"] Jul 24 03:52:20.302971 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:52:20.302931 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ce924a5_8762_48cd_a0de_29816cf6a755.slice/crio-7e02e7ca15fe3c6b1ec169ecd1c5f73bd2855315849ae2f3ef8d98207333eab4 WatchSource:0}: Error finding container 7e02e7ca15fe3c6b1ec169ecd1c5f73bd2855315849ae2f3ef8d98207333eab4: Status 404 returned error can't find the container with id 7e02e7ca15fe3c6b1ec169ecd1c5f73bd2855315849ae2f3ef8d98207333eab4 Jul 24 03:52:20.305726 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:20.305679 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:52:20.305847 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:20.305759 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:52:20.305847 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:20.305792 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:52:20.997570 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:20.997533 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" event={"ID":"0ce924a5-8762-48cd-a0de-29816cf6a755","Type":"ContainerStarted","Data":"d0196efb45058419a78f41eacd56fae95aab79bd6bc365fc4f96d04183424f7f"} Jul 24 03:52:20.997980 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:20.997589 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" event={"ID":"0ce924a5-8762-48cd-a0de-29816cf6a755","Type":"ContainerStarted","Data":"7e02e7ca15fe3c6b1ec169ecd1c5f73bd2855315849ae2f3ef8d98207333eab4"} Jul 24 03:52:21.020859 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:21.020807 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" podStartSLOduration=2.020790282 podStartE2EDuration="2.020790282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:52:21.018774798 +0000 UTC m=+2090.458194259" watchObservedRunningTime="2026-07-24 03:52:21.020790282 +0000 UTC m=+2090.460209741" Jul 24 03:52:21.161575 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:21.161538 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:21.167090 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:21.167063 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:22.002979 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:22.002938 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:22.004052 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:22.004032 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:22.927971 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:22.927938 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:23.024429 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:23.024393 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:23.127521 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:23.127440 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-676994bcf8-684nd"] Jul 24 03:52:23.128120 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:23.128098 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8843e30b-1d95-4206-a139-9b92a566dd94" containerName="istio-proxy" Jul 24 03:52:23.128120 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:23.128121 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8843e30b-1d95-4206-a139-9b92a566dd94" containerName="istio-proxy" Jul 24 03:52:23.131686 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:23.131661 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-676994bcf8-684nd" Jul 24 03:52:23.133902 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:23.133880 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-365be7db-serving-cert\"" Jul 24 03:52:23.142244 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:23.142224 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-676994bcf8-684nd"] Jul 24 03:52:23.168061 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:23.168021 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:23.291786 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:23.291707 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zxbs\" (UniqueName: \"kubernetes.io/projected/1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe-kube-api-access-6zxbs\") pod \"maas-api-e2e-mt-365be7db-676994bcf8-684nd\" (UID: \"1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-676994bcf8-684nd" Jul 24 03:52:23.291786 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:23.291741 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe-openshift-service-ca\") pod \"maas-api-e2e-mt-365be7db-676994bcf8-684nd\" (UID: \"1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-676994bcf8-684nd" Jul 24 03:52:23.291786 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:23.291787 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe-maas-api-tls\") pod \"maas-api-e2e-mt-365be7db-676994bcf8-684nd\" (UID: \"1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-676994bcf8-684nd" Jul 24 03:52:23.393254 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:23.393214 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe-maas-api-tls\") pod \"maas-api-e2e-mt-365be7db-676994bcf8-684nd\" (UID: \"1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-676994bcf8-684nd" Jul 24 03:52:23.393436 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:23.393341 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zxbs\" (UniqueName: \"kubernetes.io/projected/1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe-kube-api-access-6zxbs\") pod \"maas-api-e2e-mt-365be7db-676994bcf8-684nd\" (UID: \"1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-676994bcf8-684nd" Jul 24 03:52:23.393436 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:52:23.393365 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-serving-cert: secret "maas-api-e2e-mt-365be7db-serving-cert" not found Jul 24 03:52:23.393597 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:23.393373 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe-openshift-service-ca\") pod \"maas-api-e2e-mt-365be7db-676994bcf8-684nd\" (UID: \"1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-676994bcf8-684nd" Jul 24 03:52:23.393597 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:52:23.393443 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe-maas-api-tls podName:1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe nodeName:}" failed. No retries permitted until 2026-07-24 03:52:23.893418088 +0000 UTC m=+2093.332837527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe-maas-api-tls") pod "maas-api-e2e-mt-365be7db-676994bcf8-684nd" (UID: "1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe") : secret "maas-api-e2e-mt-365be7db-serving-cert" not found Jul 24 03:52:23.394121 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:23.394100 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe-openshift-service-ca\") pod \"maas-api-e2e-mt-365be7db-676994bcf8-684nd\" (UID: \"1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-676994bcf8-684nd" Jul 24 03:52:23.407729 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:23.407697 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zxbs\" (UniqueName: \"kubernetes.io/projected/1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe-kube-api-access-6zxbs\") pod \"maas-api-e2e-mt-365be7db-676994bcf8-684nd\" (UID: \"1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-676994bcf8-684nd" Jul 24 03:52:23.784715 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:23.784672 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:23.882440 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:23.882408 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:23.900668 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:23.900444 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe-maas-api-tls\") pod \"maas-api-e2e-mt-365be7db-676994bcf8-684nd\" (UID: \"1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-676994bcf8-684nd" Jul 24 03:52:23.903297 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:23.903276 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe-maas-api-tls\") pod \"maas-api-e2e-mt-365be7db-676994bcf8-684nd\" (UID: \"1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-676994bcf8-684nd" Jul 24 03:52:23.965563 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:23.965531 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:24.045711 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:24.045624 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-676994bcf8-684nd" Jul 24 03:52:24.203626 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:24.203598 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-676994bcf8-684nd"] Jul 24 03:52:24.205579 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:52:24.205554 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d20d50d_4bc0_4c6a_883f_6fa3588d5bfe.slice/crio-db19f0ac2b4871779f4171d04f63542c0cf8e5eb7b3fbf394ee0bb14715448f6 WatchSource:0}: Error finding container db19f0ac2b4871779f4171d04f63542c0cf8e5eb7b3fbf394ee0bb14715448f6: Status 404 returned error can't find the container with id db19f0ac2b4871779f4171d04f63542c0cf8e5eb7b3fbf394ee0bb14715448f6 Jul 24 03:52:25.019068 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:25.018976 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-676994bcf8-684nd" event={"ID":"1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe","Type":"ContainerStarted","Data":"4b58250683d7f8049251c4d99609d9a71110ebf5f9bcd73b21e084d8112abfc7"} Jul 24 03:52:25.019068 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:25.019041 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-676994bcf8-684nd" event={"ID":"1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe","Type":"ContainerStarted","Data":"db19f0ac2b4871779f4171d04f63542c0cf8e5eb7b3fbf394ee0bb14715448f6"} Jul 24 03:52:25.019254 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:25.019215 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-676994bcf8-684nd" Jul 24 03:52:25.037790 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:25.037740 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-676994bcf8-684nd" podStartSLOduration=1.5367993869999999 podStartE2EDuration="2.037726013s" totalImagesPullingTime="500.926626ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:52:25.036671911 +0000 UTC m=+2094.476091372" watchObservedRunningTime="2026-07-24 03:52:25.037726013 +0000 UTC m=+2094.477145473" Jul 24 03:52:27.427388 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:27.427355 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:28.078621 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:28.078588 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:28.178793 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:28.178758 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:28.511455 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:28.511419 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:31.035932 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:31.035904 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-676994bcf8-684nd" Jul 24 03:52:34.436803 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:34.436742 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:34.499042 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:34.499006 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:37.218992 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:37.218124 2579 patch_prober.go:28] interesting pod/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": read tcp 10.134.0.2:47884->10.134.0.58:15021: read: connection reset by peer" start-of-body= Jul 24 03:52:37.218992 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:37.218208 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" podUID="0ce924a5-8762-48cd-a0de-29816cf6a755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": read tcp 10.134.0.2:47884->10.134.0.58:15021: read: connection reset by peer" Jul 24 03:52:38.085153 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:38.085119 2579 generic.go:356] "Generic (PLEG): container finished" podID="0ce924a5-8762-48cd-a0de-29816cf6a755" containerID="d0196efb45058419a78f41eacd56fae95aab79bd6bc365fc4f96d04183424f7f" exitCode=0 Jul 24 03:52:38.085334 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:38.085192 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" event={"ID":"0ce924a5-8762-48cd-a0de-29816cf6a755","Type":"ContainerDied","Data":"d0196efb45058419a78f41eacd56fae95aab79bd6bc365fc4f96d04183424f7f"} Jul 24 03:52:38.085600 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:38.085586 2579 scope.go:117] "RemoveContainer" containerID="d0196efb45058419a78f41eacd56fae95aab79bd6bc365fc4f96d04183424f7f" Jul 24 03:52:38.086720 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:38.086689 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:52:38.086795 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:38.086780 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:52:38.086872 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:38.086854 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:52:39.095985 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:39.095932 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" event={"ID":"0ce924a5-8762-48cd-a0de-29816cf6a755","Type":"ContainerStarted","Data":"375db66f42af88e37a7c1938b48dc3b1e186ef1d4f122f4cd241c662ed4fb0d5"} Jul 24 03:52:39.161555 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:39.161519 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:39.161779 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:39.161757 2579 patch_prober.go:28] interesting pod/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 24 03:52:39.161837 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:39.161802 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" podUID="0ce924a5-8762-48cd-a0de-29816cf6a755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 24 03:52:40.162039 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:40.162002 2579 patch_prober.go:28] interesting pod/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 24 03:52:40.162445 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:40.162063 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" podUID="0ce924a5-8762-48cd-a0de-29816cf6a755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 24 03:52:41.162545 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:41.162505 2579 patch_prober.go:28] interesting pod/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 24 03:52:41.162933 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:41.162576 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" podUID="0ce924a5-8762-48cd-a0de-29816cf6a755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 24 03:52:42.162237 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:42.162201 2579 patch_prober.go:28] interesting pod/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 24 03:52:42.162414 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:42.162250 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" podUID="0ce924a5-8762-48cd-a0de-29816cf6a755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 24 03:52:43.165644 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:43.165615 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:43.166018 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:43.165972 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:43.166689 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:43.166672 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:52:45.421864 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:45.421833 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-676994bcf8-684nd"] Jul 24 03:52:45.422548 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:45.422037 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-676994bcf8-684nd" podUID="1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe" containerName="maas-api" containerID="cri-o://4b58250683d7f8049251c4d99609d9a71110ebf5f9bcd73b21e084d8112abfc7" gracePeriod=30 Jul 24 03:52:45.444479 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:45.444438 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:45.546717 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:45.546676 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:45.645208 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:45.645018 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:45.689391 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:45.689368 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-676994bcf8-684nd" Jul 24 03:52:45.748318 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:45.748283 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zxbs\" (UniqueName: \"kubernetes.io/projected/1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe-kube-api-access-6zxbs\") pod \"1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe\" (UID: \"1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe\") " Jul 24 03:52:45.748510 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:45.748328 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe-openshift-service-ca\") pod \"1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe\" (UID: \"1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe\") " Jul 24 03:52:45.748510 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:45.748350 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe-maas-api-tls\") pod \"1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe\" (UID: \"1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe\") " Jul 24 03:52:45.748735 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:45.748710 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe" (UID: "1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 03:52:45.750579 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:45.750550 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe-kube-api-access-6zxbs" (OuterVolumeSpecName: "kube-api-access-6zxbs") pod "1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe" (UID: "1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe"). InnerVolumeSpecName "kube-api-access-6zxbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:52:45.750707 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:45.750613 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe" (UID: "1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 03:52:45.849455 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:45.849423 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zxbs\" (UniqueName: \"kubernetes.io/projected/1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe-kube-api-access-6zxbs\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:52:45.849455 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:45.849450 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe-openshift-service-ca\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:52:45.849455 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:45.849487 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe-maas-api-tls\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:52:46.134819 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:46.134788 2579 generic.go:356] "Generic (PLEG): container finished" podID="1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe" containerID="4b58250683d7f8049251c4d99609d9a71110ebf5f9bcd73b21e084d8112abfc7" exitCode=0 Jul 24 03:52:46.135019 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:46.134835 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-676994bcf8-684nd" Jul 24 03:52:46.135019 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:46.134836 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-676994bcf8-684nd" event={"ID":"1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe","Type":"ContainerDied","Data":"4b58250683d7f8049251c4d99609d9a71110ebf5f9bcd73b21e084d8112abfc7"} Jul 24 03:52:46.135019 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:46.134977 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-676994bcf8-684nd" event={"ID":"1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe","Type":"ContainerDied","Data":"db19f0ac2b4871779f4171d04f63542c0cf8e5eb7b3fbf394ee0bb14715448f6"} Jul 24 03:52:46.135019 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:46.135014 2579 scope.go:117] "RemoveContainer" containerID="4b58250683d7f8049251c4d99609d9a71110ebf5f9bcd73b21e084d8112abfc7" Jul 24 03:52:46.144402 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:46.144384 2579 scope.go:117] "RemoveContainer" containerID="4b58250683d7f8049251c4d99609d9a71110ebf5f9bcd73b21e084d8112abfc7" Jul 24 03:52:46.144679 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:52:46.144663 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b58250683d7f8049251c4d99609d9a71110ebf5f9bcd73b21e084d8112abfc7\": container with ID starting with 4b58250683d7f8049251c4d99609d9a71110ebf5f9bcd73b21e084d8112abfc7 not found: ID does not exist" containerID="4b58250683d7f8049251c4d99609d9a71110ebf5f9bcd73b21e084d8112abfc7" Jul 24 03:52:46.144740 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:46.144686 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b58250683d7f8049251c4d99609d9a71110ebf5f9bcd73b21e084d8112abfc7"} err="failed to get container status \"4b58250683d7f8049251c4d99609d9a71110ebf5f9bcd73b21e084d8112abfc7\": rpc error: code = NotFound desc = could not find container \"4b58250683d7f8049251c4d99609d9a71110ebf5f9bcd73b21e084d8112abfc7\": container with ID starting with 4b58250683d7f8049251c4d99609d9a71110ebf5f9bcd73b21e084d8112abfc7 not found: ID does not exist" Jul 24 03:52:46.154892 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:46.154870 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-676994bcf8-684nd"] Jul 24 03:52:46.159221 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:46.159201 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-365be7db-676994bcf8-684nd"] Jul 24 03:52:47.033428 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:47.033394 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe" path="/var/lib/kubelet/pods/1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe/volumes" Jul 24 03:52:49.856133 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:49.856098 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:50.243747 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:50.243708 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:56.270051 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:56.270009 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:56.337965 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:56.337927 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:56.517256 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:56.517226 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln"] Jul 24 03:52:56.517535 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:56.517502 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" podUID="0ce924a5-8762-48cd-a0de-29816cf6a755" containerName="istio-proxy" containerID="cri-o://375db66f42af88e37a7c1938b48dc3b1e186ef1d4f122f4cd241c662ed4fb0d5" gracePeriod=30 Jul 24 03:52:56.543910 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:56.543830 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:56.772928 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:56.772894 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:56.887893 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:56.887822 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:56.987259 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:56.987223 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:57.100906 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:57.100872 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:52:58.168311 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:58.168275 2579 patch_prober.go:28] interesting pod/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 03:52:58.168730 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:52:58.168336 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" podUID="0ce924a5-8762-48cd-a0de-29816cf6a755" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 03:53:01.783283 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:01.783224 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:53:01.900179 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:01.900141 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ce924a5-8762-48cd-a0de-29816cf6a755-istio-token\") pod \"0ce924a5-8762-48cd-a0de-29816cf6a755\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " Jul 24 03:53:01.900377 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:01.900195 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-credential-socket\") pod \"0ce924a5-8762-48cd-a0de-29816cf6a755\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " Jul 24 03:53:01.900377 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:01.900239 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ce924a5-8762-48cd-a0de-29816cf6a755-istio-podinfo\") pod \"0ce924a5-8762-48cd-a0de-29816cf6a755\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " Jul 24 03:53:01.900377 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:01.900283 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-istio-envoy\") pod \"0ce924a5-8762-48cd-a0de-29816cf6a755\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " Jul 24 03:53:01.900377 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:01.900320 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2m9hw\" (UniqueName: \"kubernetes.io/projected/0ce924a5-8762-48cd-a0de-29816cf6a755-kube-api-access-2m9hw\") pod \"0ce924a5-8762-48cd-a0de-29816cf6a755\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " Jul 24 03:53:01.900377 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:01.900357 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-workload-socket\") pod \"0ce924a5-8762-48cd-a0de-29816cf6a755\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " Jul 24 03:53:01.900681 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:01.900389 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-istio-data\") pod \"0ce924a5-8762-48cd-a0de-29816cf6a755\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " Jul 24 03:53:01.900681 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:01.900415 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ce924a5-8762-48cd-a0de-29816cf6a755-istiod-ca-cert\") pod \"0ce924a5-8762-48cd-a0de-29816cf6a755\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " Jul 24 03:53:01.900681 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:01.900534 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-workload-certs\") pod \"0ce924a5-8762-48cd-a0de-29816cf6a755\" (UID: \"0ce924a5-8762-48cd-a0de-29816cf6a755\") " Jul 24 03:53:01.900841 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:01.900684 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0ce924a5-8762-48cd-a0de-29816cf6a755" (UID: "0ce924a5-8762-48cd-a0de-29816cf6a755"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:53:01.900933 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:01.900913 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0ce924a5-8762-48cd-a0de-29816cf6a755" (UID: "0ce924a5-8762-48cd-a0de-29816cf6a755"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:53:01.901004 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:01.900944 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-workload-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:53:01.901196 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:01.901173 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ce924a5-8762-48cd-a0de-29816cf6a755-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0ce924a5-8762-48cd-a0de-29816cf6a755" (UID: "0ce924a5-8762-48cd-a0de-29816cf6a755"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 03:53:01.901303 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:01.901221 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0ce924a5-8762-48cd-a0de-29816cf6a755" (UID: "0ce924a5-8762-48cd-a0de-29816cf6a755"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:53:01.901562 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:01.901534 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-istio-data" (OuterVolumeSpecName: "istio-data") pod "0ce924a5-8762-48cd-a0de-29816cf6a755" (UID: "0ce924a5-8762-48cd-a0de-29816cf6a755"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:53:01.902641 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:01.902617 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ce924a5-8762-48cd-a0de-29816cf6a755-istio-token" (OuterVolumeSpecName: "istio-token") pod "0ce924a5-8762-48cd-a0de-29816cf6a755" (UID: "0ce924a5-8762-48cd-a0de-29816cf6a755"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:53:01.902723 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:01.902641 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0ce924a5-8762-48cd-a0de-29816cf6a755" (UID: "0ce924a5-8762-48cd-a0de-29816cf6a755"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:53:01.903008 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:01.902985 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ce924a5-8762-48cd-a0de-29816cf6a755-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0ce924a5-8762-48cd-a0de-29816cf6a755" (UID: "0ce924a5-8762-48cd-a0de-29816cf6a755"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 03:53:01.903105 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:01.903009 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ce924a5-8762-48cd-a0de-29816cf6a755-kube-api-access-2m9hw" (OuterVolumeSpecName: "kube-api-access-2m9hw") pod "0ce924a5-8762-48cd-a0de-29816cf6a755" (UID: "0ce924a5-8762-48cd-a0de-29816cf6a755"). InnerVolumeSpecName "kube-api-access-2m9hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:53:02.001767 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:02.001731 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ce924a5-8762-48cd-a0de-29816cf6a755-istio-token\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:53:02.001767 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:02.001759 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-credential-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:53:02.001767 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:02.001769 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ce924a5-8762-48cd-a0de-29816cf6a755-istio-podinfo\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:53:02.001767 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:02.001777 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-istio-envoy\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:53:02.002017 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:02.001786 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m9hw\" (UniqueName: \"kubernetes.io/projected/0ce924a5-8762-48cd-a0de-29816cf6a755-kube-api-access-2m9hw\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:53:02.002017 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:02.001795 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-istio-data\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:53:02.002017 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:02.001803 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ce924a5-8762-48cd-a0de-29816cf6a755-istiod-ca-cert\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:53:02.002017 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:02.001812 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ce924a5-8762-48cd-a0de-29816cf6a755-workload-certs\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:53:02.217082 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:02.217044 2579 generic.go:356] "Generic (PLEG): container finished" podID="0ce924a5-8762-48cd-a0de-29816cf6a755" containerID="375db66f42af88e37a7c1938b48dc3b1e186ef1d4f122f4cd241c662ed4fb0d5" exitCode=0 Jul 24 03:53:02.217258 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:02.217112 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" Jul 24 03:53:02.217258 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:02.217129 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" event={"ID":"0ce924a5-8762-48cd-a0de-29816cf6a755","Type":"ContainerDied","Data":"375db66f42af88e37a7c1938b48dc3b1e186ef1d4f122f4cd241c662ed4fb0d5"} Jul 24 03:53:02.217258 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:02.217181 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln" event={"ID":"0ce924a5-8762-48cd-a0de-29816cf6a755","Type":"ContainerDied","Data":"7e02e7ca15fe3c6b1ec169ecd1c5f73bd2855315849ae2f3ef8d98207333eab4"} Jul 24 03:53:02.217258 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:02.217203 2579 scope.go:117] "RemoveContainer" containerID="375db66f42af88e37a7c1938b48dc3b1e186ef1d4f122f4cd241c662ed4fb0d5" Jul 24 03:53:02.226755 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:02.226735 2579 scope.go:117] "RemoveContainer" containerID="d0196efb45058419a78f41eacd56fae95aab79bd6bc365fc4f96d04183424f7f" Jul 24 03:53:02.234925 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:02.234909 2579 scope.go:117] "RemoveContainer" containerID="375db66f42af88e37a7c1938b48dc3b1e186ef1d4f122f4cd241c662ed4fb0d5" Jul 24 03:53:02.235168 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:53:02.235149 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"375db66f42af88e37a7c1938b48dc3b1e186ef1d4f122f4cd241c662ed4fb0d5\": container with ID starting with 375db66f42af88e37a7c1938b48dc3b1e186ef1d4f122f4cd241c662ed4fb0d5 not found: ID does not exist" containerID="375db66f42af88e37a7c1938b48dc3b1e186ef1d4f122f4cd241c662ed4fb0d5" Jul 24 03:53:02.235237 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:02.235182 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"375db66f42af88e37a7c1938b48dc3b1e186ef1d4f122f4cd241c662ed4fb0d5"} err="failed to get container status \"375db66f42af88e37a7c1938b48dc3b1e186ef1d4f122f4cd241c662ed4fb0d5\": rpc error: code = NotFound desc = could not find container \"375db66f42af88e37a7c1938b48dc3b1e186ef1d4f122f4cd241c662ed4fb0d5\": container with ID starting with 375db66f42af88e37a7c1938b48dc3b1e186ef1d4f122f4cd241c662ed4fb0d5 not found: ID does not exist" Jul 24 03:53:02.235237 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:02.235206 2579 scope.go:117] "RemoveContainer" containerID="d0196efb45058419a78f41eacd56fae95aab79bd6bc365fc4f96d04183424f7f" Jul 24 03:53:02.235444 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:53:02.235428 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0196efb45058419a78f41eacd56fae95aab79bd6bc365fc4f96d04183424f7f\": container with ID starting with d0196efb45058419a78f41eacd56fae95aab79bd6bc365fc4f96d04183424f7f not found: ID does not exist" containerID="d0196efb45058419a78f41eacd56fae95aab79bd6bc365fc4f96d04183424f7f" Jul 24 03:53:02.235504 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:02.235449 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0196efb45058419a78f41eacd56fae95aab79bd6bc365fc4f96d04183424f7f"} err="failed to get container status \"d0196efb45058419a78f41eacd56fae95aab79bd6bc365fc4f96d04183424f7f\": rpc error: code = NotFound desc = could not find container \"d0196efb45058419a78f41eacd56fae95aab79bd6bc365fc4f96d04183424f7f\": container with ID starting with d0196efb45058419a78f41eacd56fae95aab79bd6bc365fc4f96d04183424f7f not found: ID does not exist" Jul 24 03:53:02.238377 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:02.238355 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln"] Jul 24 03:53:02.242324 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:02.242304 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-365be7db-openshift-default-5b5cc9ccb-lphln"] Jul 24 03:53:03.034793 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:03.034760 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ce924a5-8762-48cd-a0de-29816cf6a755" path="/var/lib/kubelet/pods/0ce924a5-8762-48cd-a0de-29816cf6a755/volumes" Jul 24 03:53:03.082049 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:03.082010 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:53:03.233505 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:03.233448 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:53:03.347393 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:03.347303 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:53:03.469763 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:03.469728 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:53:09.037207 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:09.037170 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:53:09.137428 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:09.137403 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:53:15.795825 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:15.795788 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:53:16.096336 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.096266 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh"] Jul 24 03:53:16.096687 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.096674 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ce924a5-8762-48cd-a0de-29816cf6a755" containerName="istio-proxy" Jul 24 03:53:16.096744 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.096689 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ce924a5-8762-48cd-a0de-29816cf6a755" containerName="istio-proxy" Jul 24 03:53:16.096744 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.096702 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe" containerName="maas-api" Jul 24 03:53:16.096744 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.096707 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe" containerName="maas-api" Jul 24 03:53:16.096744 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.096714 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ce924a5-8762-48cd-a0de-29816cf6a755" containerName="istio-proxy" Jul 24 03:53:16.096744 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.096719 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ce924a5-8762-48cd-a0de-29816cf6a755" containerName="istio-proxy" Jul 24 03:53:16.096920 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.096789 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d20d50d-4bc0-4c6a-883f-6fa3588d5bfe" containerName="maas-api" Jul 24 03:53:16.096920 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.096799 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ce924a5-8762-48cd-a0de-29816cf6a755" containerName="istio-proxy" Jul 24 03:53:16.096920 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.096806 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ce924a5-8762-48cd-a0de-29816cf6a755" containerName="istio-proxy" Jul 24 03:53:16.099875 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.099859 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.102299 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.102280 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e2438b97-openshift-default-dockercfg-2p7jt\"" Jul 24 03:53:16.113949 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.113917 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh"] Jul 24 03:53:16.123686 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.123660 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istio-data\") pod \"e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.123811 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.123697 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-credential-socket\") pod \"e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.123811 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.123757 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kckwn\" (UniqueName: \"kubernetes.io/projected/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-kube-api-access-kckwn\") pod \"e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.123811 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.123802 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-workload-certs\") pod \"e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.123959 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.123827 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istiod-ca-cert\") pod \"e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.123959 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.123907 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istio-token\") pod \"e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.123959 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.123941 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-workload-socket\") pod \"e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.124067 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.123964 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istio-podinfo\") pod \"e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.124067 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.124032 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istio-envoy\") pod \"e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.130581 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.130531 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:53:16.224407 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.224367 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:53:16.224624 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.224477 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kckwn\" (UniqueName: \"kubernetes.io/projected/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-kube-api-access-kckwn\") pod \"e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.224624 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.224531 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-workload-certs\") pod \"e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.224755 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.224659 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istiod-ca-cert\") pod \"e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.224755 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.224707 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istio-token\") pod \"e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.224863 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.224754 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-workload-socket\") pod \"e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.224863 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.224784 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istio-podinfo\") pod \"e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.224863 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.224849 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istio-envoy\") pod \"e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.225020 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.224899 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-workload-certs\") pod \"e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.225020 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.224905 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istio-data\") pod \"e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.225020 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.224932 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-credential-socket\") pod \"e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.225178 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.225164 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-workload-socket\") pod \"e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.225310 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.225288 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-credential-socket\") pod \"e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.225381 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.225348 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istiod-ca-cert\") pod \"e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.225446 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.225412 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istio-data\") pod \"e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.227572 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.227496 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istio-envoy\") pod \"e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.228020 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.227997 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istio-podinfo\") pod \"e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.234281 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.234257 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kckwn\" (UniqueName: \"kubernetes.io/projected/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-kube-api-access-kckwn\") pod \"e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.234433 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.234354 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istio-token\") pod \"e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.411746 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.411706 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:16.554237 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.554207 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh"] Jul 24 03:53:16.555762 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:53:16.555735 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a17cdb_7bb2_4e38_8826_1bc1ab862c6c.slice/crio-d6e816d6b558e44a5bcd05a9c3a12c68b56b20f85b59bbaf9d0fbdb3e4aacdc5 WatchSource:0}: Error finding container d6e816d6b558e44a5bcd05a9c3a12c68b56b20f85b59bbaf9d0fbdb3e4aacdc5: Status 404 returned error can't find the container with id d6e816d6b558e44a5bcd05a9c3a12c68b56b20f85b59bbaf9d0fbdb3e4aacdc5 Jul 24 03:53:16.558023 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.557987 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:53:16.558132 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.558069 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:53:16.558132 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:16.558117 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:53:17.301517 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:17.301486 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" event={"ID":"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c","Type":"ContainerStarted","Data":"7e2eae797490cc36e4400aee69f856c400e5ca8035d04c368abf55642a93415b"} Jul 24 03:53:17.301896 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:17.301536 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" event={"ID":"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c","Type":"ContainerStarted","Data":"d6e816d6b558e44a5bcd05a9c3a12c68b56b20f85b59bbaf9d0fbdb3e4aacdc5"} Jul 24 03:53:17.322879 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:17.322829 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" podStartSLOduration=1.322813255 podStartE2EDuration="1.322813255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:53:17.320492572 +0000 UTC m=+2146.759912029" watchObservedRunningTime="2026-07-24 03:53:17.322813255 +0000 UTC m=+2146.762232714" Jul 24 03:53:17.412232 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:17.412199 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:17.418024 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:17.417998 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:18.306270 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:18.306241 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:18.307279 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:18.307257 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:19.229628 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:19.229587 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:53:19.299932 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:19.299898 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-559699465d-58g52"] Jul 24 03:53:19.303298 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:19.303272 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-559699465d-58g52" Jul 24 03:53:19.303962 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:19.303937 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:53:19.305576 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:19.305558 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e2438b97-serving-cert\"" Jul 24 03:53:19.313579 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:19.313347 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-559699465d-58g52"] Jul 24 03:53:19.356388 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:19.356347 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/997c48e8-cabd-47c9-88e3-2b08f2ab4741-openshift-service-ca\") pod \"maas-api-e2e-mt-e2438b97-559699465d-58g52\" (UID: \"997c48e8-cabd-47c9-88e3-2b08f2ab4741\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-559699465d-58g52" Jul 24 03:53:19.356600 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:19.356405 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7rrl\" (UniqueName: \"kubernetes.io/projected/997c48e8-cabd-47c9-88e3-2b08f2ab4741-kube-api-access-v7rrl\") pod \"maas-api-e2e-mt-e2438b97-559699465d-58g52\" (UID: \"997c48e8-cabd-47c9-88e3-2b08f2ab4741\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-559699465d-58g52" Jul 24 03:53:19.356600 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:19.356445 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/997c48e8-cabd-47c9-88e3-2b08f2ab4741-maas-api-tls\") pod \"maas-api-e2e-mt-e2438b97-559699465d-58g52\" (UID: \"997c48e8-cabd-47c9-88e3-2b08f2ab4741\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-559699465d-58g52" Jul 24 03:53:19.380834 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:19.380801 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:53:19.457410 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:19.457376 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/997c48e8-cabd-47c9-88e3-2b08f2ab4741-maas-api-tls\") pod \"maas-api-e2e-mt-e2438b97-559699465d-58g52\" (UID: \"997c48e8-cabd-47c9-88e3-2b08f2ab4741\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-559699465d-58g52" Jul 24 03:53:19.457625 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:19.457496 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/997c48e8-cabd-47c9-88e3-2b08f2ab4741-openshift-service-ca\") pod \"maas-api-e2e-mt-e2438b97-559699465d-58g52\" (UID: \"997c48e8-cabd-47c9-88e3-2b08f2ab4741\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-559699465d-58g52" Jul 24 03:53:19.457625 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:19.457543 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7rrl\" (UniqueName: \"kubernetes.io/projected/997c48e8-cabd-47c9-88e3-2b08f2ab4741-kube-api-access-v7rrl\") pod \"maas-api-e2e-mt-e2438b97-559699465d-58g52\" (UID: \"997c48e8-cabd-47c9-88e3-2b08f2ab4741\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-559699465d-58g52" Jul 24 03:53:19.457625 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:53:19.457562 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-serving-cert: secret "maas-api-e2e-mt-e2438b97-serving-cert" not found Jul 24 03:53:19.457752 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:53:19.457643 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/997c48e8-cabd-47c9-88e3-2b08f2ab4741-maas-api-tls podName:997c48e8-cabd-47c9-88e3-2b08f2ab4741 nodeName:}" failed. No retries permitted until 2026-07-24 03:53:19.957622923 +0000 UTC m=+2149.397042361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/997c48e8-cabd-47c9-88e3-2b08f2ab4741-maas-api-tls") pod "maas-api-e2e-mt-e2438b97-559699465d-58g52" (UID: "997c48e8-cabd-47c9-88e3-2b08f2ab4741") : secret "maas-api-e2e-mt-e2438b97-serving-cert" not found Jul 24 03:53:19.458284 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:19.458251 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/997c48e8-cabd-47c9-88e3-2b08f2ab4741-openshift-service-ca\") pod \"maas-api-e2e-mt-e2438b97-559699465d-58g52\" (UID: \"997c48e8-cabd-47c9-88e3-2b08f2ab4741\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-559699465d-58g52" Jul 24 03:53:19.471008 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:19.470986 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7rrl\" (UniqueName: \"kubernetes.io/projected/997c48e8-cabd-47c9-88e3-2b08f2ab4741-kube-api-access-v7rrl\") pod \"maas-api-e2e-mt-e2438b97-559699465d-58g52\" (UID: \"997c48e8-cabd-47c9-88e3-2b08f2ab4741\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-559699465d-58g52" Jul 24 03:53:19.585555 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:19.585432 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:53:19.874885 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:19.874848 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:53:19.961531 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:19.961501 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/997c48e8-cabd-47c9-88e3-2b08f2ab4741-maas-api-tls\") pod \"maas-api-e2e-mt-e2438b97-559699465d-58g52\" (UID: \"997c48e8-cabd-47c9-88e3-2b08f2ab4741\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-559699465d-58g52" Jul 24 03:53:19.964108 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:19.964077 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/997c48e8-cabd-47c9-88e3-2b08f2ab4741-maas-api-tls\") pod \"maas-api-e2e-mt-e2438b97-559699465d-58g52\" (UID: \"997c48e8-cabd-47c9-88e3-2b08f2ab4741\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-559699465d-58g52" Jul 24 03:53:19.973010 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:19.972941 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:53:20.217730 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:20.217639 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-559699465d-58g52" Jul 24 03:53:20.221488 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:20.220898 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:53:20.366889 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:20.366864 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-559699465d-58g52"] Jul 24 03:53:20.368413 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:53:20.368389 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997c48e8_cabd_47c9_88e3_2b08f2ab4741.slice/crio-edfb2f6cadaf589961a3d2206ae5785d09bb728ccfeb3d8df032d32b90f7d3a7 WatchSource:0}: Error finding container edfb2f6cadaf589961a3d2206ae5785d09bb728ccfeb3d8df032d32b90f7d3a7: Status 404 returned error can't find the container with id edfb2f6cadaf589961a3d2206ae5785d09bb728ccfeb3d8df032d32b90f7d3a7 Jul 24 03:53:21.328932 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:21.328892 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-559699465d-58g52" event={"ID":"997c48e8-cabd-47c9-88e3-2b08f2ab4741","Type":"ContainerStarted","Data":"f6699ee669f0e1976163ac6de46c6799b8c4a9eb4854f32e9452ac245a60bb94"} Jul 24 03:53:21.329134 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:21.328947 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-559699465d-58g52" event={"ID":"997c48e8-cabd-47c9-88e3-2b08f2ab4741","Type":"ContainerStarted","Data":"edfb2f6cadaf589961a3d2206ae5785d09bb728ccfeb3d8df032d32b90f7d3a7"} Jul 24 03:53:21.329134 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:21.329020 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-559699465d-58g52" Jul 24 03:53:21.347854 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:21.347803 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-559699465d-58g52" podStartSLOduration=2.010004469 podStartE2EDuration="2.347787547s" totalImagesPullingTime="337.783078ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:53:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:53:21.346299687 +0000 UTC m=+2150.785719161" watchObservedRunningTime="2026-07-24 03:53:21.347787547 +0000 UTC m=+2150.787207024" Jul 24 03:53:23.830959 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:23.830863 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:53:27.339663 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:27.339636 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-559699465d-58g52" Jul 24 03:53:34.004962 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:34.003965 2579 patch_prober.go:28] interesting pod/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": read tcp 10.134.0.2:50216->10.134.0.60:15021: read: connection reset by peer" start-of-body= Jul 24 03:53:34.004962 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:34.004036 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" podUID="e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": read tcp 10.134.0.2:50216->10.134.0.60:15021: read: connection reset by peer" Jul 24 03:53:34.401094 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:34.401063 2579 generic.go:356] "Generic (PLEG): container finished" podID="e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c" containerID="7e2eae797490cc36e4400aee69f856c400e5ca8035d04c368abf55642a93415b" exitCode=0 Jul 24 03:53:34.401294 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:34.401143 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" event={"ID":"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c","Type":"ContainerDied","Data":"7e2eae797490cc36e4400aee69f856c400e5ca8035d04c368abf55642a93415b"} Jul 24 03:53:34.401624 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:34.401605 2579 scope.go:117] "RemoveContainer" containerID="7e2eae797490cc36e4400aee69f856c400e5ca8035d04c368abf55642a93415b" Jul 24 03:53:34.402687 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:34.402661 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:53:34.402749 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:34.402730 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:53:34.402786 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:34.402761 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:53:35.408625 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:35.408588 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" event={"ID":"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c","Type":"ContainerStarted","Data":"31db42cab882de48bd5796f19b11f82eafece186f0938ec7834d83b72c250955"} Jul 24 03:53:35.411861 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:35.411836 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:35.412037 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:35.412016 2579 patch_prober.go:28] interesting pod/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 24 03:53:35.412110 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:35.412069 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" podUID="e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 24 03:53:36.412439 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:36.412404 2579 patch_prober.go:28] interesting pod/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 24 03:53:36.412848 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:36.412480 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" podUID="e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 24 03:53:37.413040 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:37.413007 2579 patch_prober.go:28] interesting pod/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 24 03:53:37.413425 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:37.413061 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" podUID="e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 24 03:53:38.412097 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:38.412063 2579 patch_prober.go:28] interesting pod/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 24 03:53:38.412305 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:38.412147 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" podUID="e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 24 03:53:39.416237 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:39.416202 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:39.416634 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:39.416513 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:39.417217 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:39.417199 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:53:42.352428 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:53:42.352396 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:54:00.552251 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:00.552221 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:54:00.653416 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:00.653383 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:54:18.738962 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:18.738922 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-559699465d-58g52"] Jul 24 03:54:18.739419 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:18.739146 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-559699465d-58g52" podUID="997c48e8-cabd-47c9-88e3-2b08f2ab4741" containerName="maas-api" containerID="cri-o://f6699ee669f0e1976163ac6de46c6799b8c4a9eb4854f32e9452ac245a60bb94" gracePeriod=30 Jul 24 03:54:18.766285 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:18.766254 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:54:18.863301 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:18.863256 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:54:18.997871 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:18.997809 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-559699465d-58g52" Jul 24 03:54:19.132305 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:19.132271 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7rrl\" (UniqueName: \"kubernetes.io/projected/997c48e8-cabd-47c9-88e3-2b08f2ab4741-kube-api-access-v7rrl\") pod \"997c48e8-cabd-47c9-88e3-2b08f2ab4741\" (UID: \"997c48e8-cabd-47c9-88e3-2b08f2ab4741\") " Jul 24 03:54:19.132528 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:19.132388 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/997c48e8-cabd-47c9-88e3-2b08f2ab4741-maas-api-tls\") pod \"997c48e8-cabd-47c9-88e3-2b08f2ab4741\" (UID: \"997c48e8-cabd-47c9-88e3-2b08f2ab4741\") " Jul 24 03:54:19.132528 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:19.132432 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/997c48e8-cabd-47c9-88e3-2b08f2ab4741-openshift-service-ca\") pod \"997c48e8-cabd-47c9-88e3-2b08f2ab4741\" (UID: \"997c48e8-cabd-47c9-88e3-2b08f2ab4741\") " Jul 24 03:54:19.132836 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:19.132807 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/997c48e8-cabd-47c9-88e3-2b08f2ab4741-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "997c48e8-cabd-47c9-88e3-2b08f2ab4741" (UID: "997c48e8-cabd-47c9-88e3-2b08f2ab4741"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 03:54:19.133053 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:19.133003 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/997c48e8-cabd-47c9-88e3-2b08f2ab4741-openshift-service-ca\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:54:19.134816 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:19.134788 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/997c48e8-cabd-47c9-88e3-2b08f2ab4741-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "997c48e8-cabd-47c9-88e3-2b08f2ab4741" (UID: "997c48e8-cabd-47c9-88e3-2b08f2ab4741"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 03:54:19.134908 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:19.134891 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/997c48e8-cabd-47c9-88e3-2b08f2ab4741-kube-api-access-v7rrl" (OuterVolumeSpecName: "kube-api-access-v7rrl") pod "997c48e8-cabd-47c9-88e3-2b08f2ab4741" (UID: "997c48e8-cabd-47c9-88e3-2b08f2ab4741"). InnerVolumeSpecName "kube-api-access-v7rrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:54:19.233907 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:19.233877 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7rrl\" (UniqueName: \"kubernetes.io/projected/997c48e8-cabd-47c9-88e3-2b08f2ab4741-kube-api-access-v7rrl\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:54:19.233907 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:19.233902 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/997c48e8-cabd-47c9-88e3-2b08f2ab4741-maas-api-tls\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:54:19.631708 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:19.631678 2579 generic.go:356] "Generic (PLEG): container finished" podID="997c48e8-cabd-47c9-88e3-2b08f2ab4741" containerID="f6699ee669f0e1976163ac6de46c6799b8c4a9eb4854f32e9452ac245a60bb94" exitCode=0 Jul 24 03:54:19.631876 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:19.631731 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-559699465d-58g52" Jul 24 03:54:19.631876 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:19.631751 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-559699465d-58g52" event={"ID":"997c48e8-cabd-47c9-88e3-2b08f2ab4741","Type":"ContainerDied","Data":"f6699ee669f0e1976163ac6de46c6799b8c4a9eb4854f32e9452ac245a60bb94"} Jul 24 03:54:19.631876 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:19.631795 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-559699465d-58g52" event={"ID":"997c48e8-cabd-47c9-88e3-2b08f2ab4741","Type":"ContainerDied","Data":"edfb2f6cadaf589961a3d2206ae5785d09bb728ccfeb3d8df032d32b90f7d3a7"} Jul 24 03:54:19.631876 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:19.631817 2579 scope.go:117] "RemoveContainer" containerID="f6699ee669f0e1976163ac6de46c6799b8c4a9eb4854f32e9452ac245a60bb94" Jul 24 03:54:19.641629 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:19.641610 2579 scope.go:117] "RemoveContainer" containerID="f6699ee669f0e1976163ac6de46c6799b8c4a9eb4854f32e9452ac245a60bb94" Jul 24 03:54:19.641908 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:54:19.641890 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6699ee669f0e1976163ac6de46c6799b8c4a9eb4854f32e9452ac245a60bb94\": container with ID starting with f6699ee669f0e1976163ac6de46c6799b8c4a9eb4854f32e9452ac245a60bb94 not found: ID does not exist" containerID="f6699ee669f0e1976163ac6de46c6799b8c4a9eb4854f32e9452ac245a60bb94" Jul 24 03:54:19.641967 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:19.641914 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6699ee669f0e1976163ac6de46c6799b8c4a9eb4854f32e9452ac245a60bb94"} err="failed to get container status \"f6699ee669f0e1976163ac6de46c6799b8c4a9eb4854f32e9452ac245a60bb94\": rpc error: code = NotFound desc = could not find container \"f6699ee669f0e1976163ac6de46c6799b8c4a9eb4854f32e9452ac245a60bb94\": container with ID starting with f6699ee669f0e1976163ac6de46c6799b8c4a9eb4854f32e9452ac245a60bb94 not found: ID does not exist" Jul 24 03:54:19.653424 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:19.653400 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-559699465d-58g52"] Jul 24 03:54:19.657023 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:19.657002 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2438b97-559699465d-58g52"] Jul 24 03:54:21.034425 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:21.034395 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="997c48e8-cabd-47c9-88e3-2b08f2ab4741" path="/var/lib/kubelet/pods/997c48e8-cabd-47c9-88e3-2b08f2ab4741/volumes" Jul 24 03:54:23.321149 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:23.321116 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:54:26.536122 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:26.536076 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:54:31.617747 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:31.617693 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:54:39.427239 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:54:39.427200 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:55:03.109249 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:03.109168 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:55:09.192681 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:09.192645 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:55:09.522231 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:09.522154 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:55:15.017623 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:15.017591 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:55:20.791210 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:20.791180 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:55:20.891761 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:20.891714 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:55:21.070228 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:21.070113 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh"] Jul 24 03:55:21.070425 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:21.070318 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" podUID="e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c" containerName="istio-proxy" containerID="cri-o://31db42cab882de48bd5796f19b11f82eafece186f0938ec7834d83b72c250955" gracePeriod=30 Jul 24 03:55:21.099475 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:21.099441 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:55:21.301899 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:21.301865 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:55:22.371035 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:22.370998 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:55:22.468887 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:22.468859 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:55:22.569912 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:22.569882 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:55:23.343907 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:23.343868 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:55:23.477936 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:23.477902 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:55:23.586613 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:23.586580 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:55:24.419573 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:24.419534 2579 patch_prober.go:28] interesting pod/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 03:55:24.419733 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:24.419607 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" podUID="e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 03:55:24.975918 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:24.975885 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:55:26.338752 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.338695 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:55:26.359331 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.359253 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-workload-socket\") pod \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " Jul 24 03:55:26.359331 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.359318 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istiod-ca-cert\") pod \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " Jul 24 03:55:26.359585 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.359348 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istio-token\") pod \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " Jul 24 03:55:26.359585 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.359416 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istio-podinfo\") pod \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " Jul 24 03:55:26.359585 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.359444 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-credential-socket\") pod \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " Jul 24 03:55:26.359585 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.359525 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istio-envoy\") pod \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " Jul 24 03:55:26.359585 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.359556 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kckwn\" (UniqueName: \"kubernetes.io/projected/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-kube-api-access-kckwn\") pod \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " Jul 24 03:55:26.360048 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.359602 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istio-data\") pod \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " Jul 24 03:55:26.360048 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.359657 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-workload-certs\") pod \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\" (UID: \"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c\") " Jul 24 03:55:26.360164 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.359551 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c" (UID: "e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:55:26.360164 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.360065 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c" (UID: "e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:55:26.361171 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.360295 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c" (UID: "e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:55:26.361171 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.360372 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c" (UID: "e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 03:55:26.361379 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.361343 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istio-data" (OuterVolumeSpecName: "istio-data") pod "e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c" (UID: "e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:55:26.362235 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.362203 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c" (UID: "e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 03:55:26.362922 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.362889 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istio-token" (OuterVolumeSpecName: "istio-token") pod "e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c" (UID: "e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:55:26.363257 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.363221 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-kube-api-access-kckwn" (OuterVolumeSpecName: "kube-api-access-kckwn") pod "e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c" (UID: "e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c"). InnerVolumeSpecName "kube-api-access-kckwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:55:26.364041 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.364020 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c" (UID: "e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:55:26.460452 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.460424 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istio-podinfo\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:55:26.460452 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.460448 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-credential-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:55:26.460663 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.460478 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istio-envoy\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:55:26.460663 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.460487 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kckwn\" (UniqueName: \"kubernetes.io/projected/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-kube-api-access-kckwn\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:55:26.460663 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.460497 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istio-data\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:55:26.460663 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.460505 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-workload-certs\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:55:26.460663 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.460514 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-workload-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:55:26.460663 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.460524 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istiod-ca-cert\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:55:26.460663 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.460531 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c-istio-token\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:55:26.980020 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.979989 2579 generic.go:356] "Generic (PLEG): container finished" podID="e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c" containerID="31db42cab882de48bd5796f19b11f82eafece186f0938ec7834d83b72c250955" exitCode=0 Jul 24 03:55:26.980248 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.980044 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" Jul 24 03:55:26.980248 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.980072 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" event={"ID":"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c","Type":"ContainerDied","Data":"31db42cab882de48bd5796f19b11f82eafece186f0938ec7834d83b72c250955"} Jul 24 03:55:26.980248 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.980131 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh" event={"ID":"e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c","Type":"ContainerDied","Data":"d6e816d6b558e44a5bcd05a9c3a12c68b56b20f85b59bbaf9d0fbdb3e4aacdc5"} Jul 24 03:55:26.980248 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.980161 2579 scope.go:117] "RemoveContainer" containerID="31db42cab882de48bd5796f19b11f82eafece186f0938ec7834d83b72c250955" Jul 24 03:55:26.991262 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.991243 2579 scope.go:117] "RemoveContainer" containerID="7e2eae797490cc36e4400aee69f856c400e5ca8035d04c368abf55642a93415b" Jul 24 03:55:26.999694 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.999673 2579 scope.go:117] "RemoveContainer" containerID="31db42cab882de48bd5796f19b11f82eafece186f0938ec7834d83b72c250955" Jul 24 03:55:26.999941 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:55:26.999924 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31db42cab882de48bd5796f19b11f82eafece186f0938ec7834d83b72c250955\": container with ID starting with 31db42cab882de48bd5796f19b11f82eafece186f0938ec7834d83b72c250955 not found: ID does not exist" containerID="31db42cab882de48bd5796f19b11f82eafece186f0938ec7834d83b72c250955" Jul 24 03:55:27.000009 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.999949 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31db42cab882de48bd5796f19b11f82eafece186f0938ec7834d83b72c250955"} err="failed to get container status \"31db42cab882de48bd5796f19b11f82eafece186f0938ec7834d83b72c250955\": rpc error: code = NotFound desc = could not find container \"31db42cab882de48bd5796f19b11f82eafece186f0938ec7834d83b72c250955\": container with ID starting with 31db42cab882de48bd5796f19b11f82eafece186f0938ec7834d83b72c250955 not found: ID does not exist" Jul 24 03:55:27.000009 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:26.999967 2579 scope.go:117] "RemoveContainer" containerID="7e2eae797490cc36e4400aee69f856c400e5ca8035d04c368abf55642a93415b" Jul 24 03:55:27.000226 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:55:27.000185 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e2eae797490cc36e4400aee69f856c400e5ca8035d04c368abf55642a93415b\": container with ID starting with 7e2eae797490cc36e4400aee69f856c400e5ca8035d04c368abf55642a93415b not found: ID does not exist" containerID="7e2eae797490cc36e4400aee69f856c400e5ca8035d04c368abf55642a93415b" Jul 24 03:55:27.000226 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:27.000208 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e2eae797490cc36e4400aee69f856c400e5ca8035d04c368abf55642a93415b"} err="failed to get container status \"7e2eae797490cc36e4400aee69f856c400e5ca8035d04c368abf55642a93415b\": rpc error: code = NotFound desc = could not find container \"7e2eae797490cc36e4400aee69f856c400e5ca8035d04c368abf55642a93415b\": container with ID starting with 7e2eae797490cc36e4400aee69f856c400e5ca8035d04c368abf55642a93415b not found: ID does not exist" Jul 24 03:55:27.002983 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:27.002961 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh"] Jul 24 03:55:27.008331 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:27.008310 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e2438b97-openshift-default-664bfbb94d-dwrmh"] Jul 24 03:55:27.033879 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:27.033849 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c" path="/var/lib/kubelet/pods/e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c/volumes" Jul 24 03:55:31.100536 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:31.100501 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:55:31.198373 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:31.198338 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:55:31.798811 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:31.798774 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:55:38.104276 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:38.104243 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:55:38.210527 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:38.210488 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:55:38.304704 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:38.304648 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:55:48.897110 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:48.897075 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:55:49.201824 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:55:49.201713 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:56:10.165621 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:10.165584 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:56:16.228081 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:16.228045 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:56:16.329664 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:16.329629 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:56:16.583502 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:16.583399 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:56:16.857546 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:16.857524 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w"] Jul 24 03:56:16.858171 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:16.858147 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c" containerName="istio-proxy" Jul 24 03:56:16.858171 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:16.858169 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c" containerName="istio-proxy" Jul 24 03:56:16.858337 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:16.858186 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="997c48e8-cabd-47c9-88e3-2b08f2ab4741" containerName="maas-api" Jul 24 03:56:16.858337 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:16.858201 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="997c48e8-cabd-47c9-88e3-2b08f2ab4741" containerName="maas-api" Jul 24 03:56:16.858337 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:16.858209 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c" containerName="istio-proxy" Jul 24 03:56:16.858337 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:16.858214 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c" containerName="istio-proxy" Jul 24 03:56:16.858337 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:16.858275 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="997c48e8-cabd-47c9-88e3-2b08f2ab4741" containerName="maas-api" Jul 24 03:56:16.858337 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:16.858285 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c" containerName="istio-proxy" Jul 24 03:56:16.858337 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:16.858292 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3a17cdb-7bb2-4e38-8826-1bc1ab862c6c" containerName="istio-proxy" Jul 24 03:56:16.861361 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:16.861345 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:16.863702 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:16.863680 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-93dff3-openshift-default-dockercfg-7vcjr\"" Jul 24 03:56:16.872070 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:16.872049 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w"] Jul 24 03:56:16.885631 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:16.885602 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:56:16.942716 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:16.942685 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw9c8\" (UniqueName: \"kubernetes.io/projected/32705bc1-1bbd-43fe-a689-611ea28a7e69-kube-api-access-jw9c8\") pod \"e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:16.942870 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:16.942732 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32705bc1-1bbd-43fe-a689-611ea28a7e69-istio-token\") pod \"e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:16.942870 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:16.942753 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-workload-socket\") pod \"e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:16.942870 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:16.942775 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-credential-socket\") pod \"e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:16.942870 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:16.942799 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-istio-envoy\") pod \"e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:16.942870 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:16.942846 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32705bc1-1bbd-43fe-a689-611ea28a7e69-istiod-ca-cert\") pod \"e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:16.942870 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:16.942867 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-istio-data\") pod \"e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:16.943095 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:16.942885 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32705bc1-1bbd-43fe-a689-611ea28a7e69-istio-podinfo\") pod \"e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:16.943095 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:16.942973 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-workload-certs\") pod \"e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:17.044037 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:17.044002 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-workload-certs\") pod \"e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:17.044262 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:17.044046 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw9c8\" (UniqueName: \"kubernetes.io/projected/32705bc1-1bbd-43fe-a689-611ea28a7e69-kube-api-access-jw9c8\") pod \"e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:17.044262 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:17.044172 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32705bc1-1bbd-43fe-a689-611ea28a7e69-istio-token\") pod \"e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:17.044262 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:17.044210 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-workload-socket\") pod \"e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:17.044262 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:17.044254 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-credential-socket\") pod \"e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:17.044518 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:17.044278 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-istio-envoy\") pod \"e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:17.044518 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:17.044353 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32705bc1-1bbd-43fe-a689-611ea28a7e69-istiod-ca-cert\") pod \"e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:17.044518 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:17.044389 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-istio-data\") pod \"e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:17.044518 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:17.044419 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32705bc1-1bbd-43fe-a689-611ea28a7e69-istio-podinfo\") pod \"e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:17.044518 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:17.044433 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-workload-certs\") pod \"e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:17.044723 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:17.044583 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-workload-socket\") pod \"e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:17.044723 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:17.044703 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-credential-socket\") pod \"e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:17.044822 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:17.044801 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-istio-data\") pod \"e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:17.045075 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:17.045057 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32705bc1-1bbd-43fe-a689-611ea28a7e69-istiod-ca-cert\") pod \"e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:17.046914 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:17.046892 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-istio-envoy\") pod \"e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:17.047049 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:17.047031 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32705bc1-1bbd-43fe-a689-611ea28a7e69-istio-podinfo\") pod \"e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:17.058059 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:17.058031 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32705bc1-1bbd-43fe-a689-611ea28a7e69-istio-token\") pod \"e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:17.058349 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:17.058326 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw9c8\" (UniqueName: \"kubernetes.io/projected/32705bc1-1bbd-43fe-a689-611ea28a7e69-kube-api-access-jw9c8\") pod \"e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:17.104581 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:17.104556 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:56:17.172189 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:17.172112 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:17.311523 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:17.311500 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w"] Jul 24 03:56:17.313369 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:56:17.313345 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32705bc1_1bbd_43fe_a689_611ea28a7e69.slice/crio-91d58ce680f3999c401b15f61fe5f0d18b1d125ecc65ac9ca0bb5a0bcfea88cf WatchSource:0}: Error finding container 91d58ce680f3999c401b15f61fe5f0d18b1d125ecc65ac9ca0bb5a0bcfea88cf: Status 404 returned error can't find the container with id 91d58ce680f3999c401b15f61fe5f0d18b1d125ecc65ac9ca0bb5a0bcfea88cf Jul 24 03:56:17.315490 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:17.315428 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:56:17.315576 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:17.315511 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:56:17.315576 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:17.315539 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:56:18.237960 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:18.237928 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" event={"ID":"32705bc1-1bbd-43fe-a689-611ea28a7e69","Type":"ContainerStarted","Data":"0548f54b379d1b64e5548fca65b4b2ca0bf3b020e22a578461c3e67975b226a4"} Jul 24 03:56:18.238133 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:18.237981 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" event={"ID":"32705bc1-1bbd-43fe-a689-611ea28a7e69","Type":"ContainerStarted","Data":"91d58ce680f3999c401b15f61fe5f0d18b1d125ecc65ac9ca0bb5a0bcfea88cf"} Jul 24 03:56:18.257785 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:18.257728 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" podStartSLOduration=2.257710028 podStartE2EDuration="2.257710028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:56:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:56:18.257092335 +0000 UTC m=+2327.696511799" watchObservedRunningTime="2026-07-24 03:56:18.257710028 +0000 UTC m=+2327.697129494" Jul 24 03:56:19.172622 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:19.172580 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:19.177861 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:19.177836 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:19.243202 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:19.243171 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:19.244207 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:19.244189 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:20.212061 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:20.212025 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:56:20.309137 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:20.309099 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:56:20.442270 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:20.442238 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq"] Jul 24 03:56:20.445808 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:20.445787 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq" Jul 24 03:56:20.448223 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:20.448198 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-93dff3-serving-cert\"" Jul 24 03:56:20.458950 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:20.458917 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq"] Jul 24 03:56:20.483141 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:20.483054 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:56:20.584542 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:20.584492 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8zmh\" (UniqueName: \"kubernetes.io/projected/e2bdd23d-e644-4b6a-a347-f74eae14047a-kube-api-access-j8zmh\") pod \"maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq\" (UID: \"e2bdd23d-e644-4b6a-a347-f74eae14047a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq" Jul 24 03:56:20.584772 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:20.584623 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2bdd23d-e644-4b6a-a347-f74eae14047a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq\" (UID: \"e2bdd23d-e644-4b6a-a347-f74eae14047a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq" Jul 24 03:56:20.584772 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:20.584691 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2bdd23d-e644-4b6a-a347-f74eae14047a-maas-api-tls\") pod \"maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq\" (UID: \"e2bdd23d-e644-4b6a-a347-f74eae14047a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq" Jul 24 03:56:20.635434 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:20.635383 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:56:20.685615 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:20.685583 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8zmh\" (UniqueName: \"kubernetes.io/projected/e2bdd23d-e644-4b6a-a347-f74eae14047a-kube-api-access-j8zmh\") pod \"maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq\" (UID: \"e2bdd23d-e644-4b6a-a347-f74eae14047a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq" Jul 24 03:56:20.685824 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:20.685644 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2bdd23d-e644-4b6a-a347-f74eae14047a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq\" (UID: \"e2bdd23d-e644-4b6a-a347-f74eae14047a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq" Jul 24 03:56:20.685824 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:20.685690 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2bdd23d-e644-4b6a-a347-f74eae14047a-maas-api-tls\") pod \"maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq\" (UID: \"e2bdd23d-e644-4b6a-a347-f74eae14047a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq" Jul 24 03:56:20.686285 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:20.686259 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2bdd23d-e644-4b6a-a347-f74eae14047a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq\" (UID: \"e2bdd23d-e644-4b6a-a347-f74eae14047a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq" Jul 24 03:56:20.688440 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:20.688421 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2bdd23d-e644-4b6a-a347-f74eae14047a-maas-api-tls\") pod \"maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq\" (UID: \"e2bdd23d-e644-4b6a-a347-f74eae14047a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq" Jul 24 03:56:20.695188 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:20.695168 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8zmh\" (UniqueName: \"kubernetes.io/projected/e2bdd23d-e644-4b6a-a347-f74eae14047a-kube-api-access-j8zmh\") pod \"maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq\" (UID: \"e2bdd23d-e644-4b6a-a347-f74eae14047a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq" Jul 24 03:56:20.757307 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:20.757221 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq" Jul 24 03:56:20.903854 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:20.903827 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq"] Jul 24 03:56:20.906277 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:56:20.906244 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2bdd23d_e644_4b6a_a347_f74eae14047a.slice/crio-99933aceb01abde773f8d87f12aa02c76de4578eb0d915469cbb07c51cbd3aaa WatchSource:0}: Error finding container 99933aceb01abde773f8d87f12aa02c76de4578eb0d915469cbb07c51cbd3aaa: Status 404 returned error can't find the container with id 99933aceb01abde773f8d87f12aa02c76de4578eb0d915469cbb07c51cbd3aaa Jul 24 03:56:20.907811 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:20.907791 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 03:56:21.066499 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:21.066410 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:56:21.164970 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:21.164932 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:56:21.255908 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:21.255871 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq" event={"ID":"e2bdd23d-e644-4b6a-a347-f74eae14047a","Type":"ContainerStarted","Data":"99933aceb01abde773f8d87f12aa02c76de4578eb0d915469cbb07c51cbd3aaa"} Jul 24 03:56:22.263497 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:22.263381 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq" event={"ID":"e2bdd23d-e644-4b6a-a347-f74eae14047a","Type":"ContainerStarted","Data":"e818121e6420e183962ee3dd8ae571cd472dd5f6492982c82bd3719f952c892e"} Jul 24 03:56:22.263971 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:22.263559 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq" Jul 24 03:56:22.280146 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:22.280098 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq" podStartSLOduration=1.925505656 podStartE2EDuration="2.280078997s" totalImagesPullingTime="354.573341ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:56:22.279514145 +0000 UTC m=+2331.718933606" watchObservedRunningTime="2026-07-24 03:56:22.280078997 +0000 UTC m=+2331.719498459" Jul 24 03:56:24.704414 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:24.704379 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:56:25.054118 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.054038 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:56:25.301090 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.301059 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc"] Jul 24 03:56:25.305196 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.305131 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.307502 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.307481 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b0d347-openshift-default-dockercfg-vlnvx\"" Jul 24 03:56:25.317123 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.317097 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc"] Jul 24 03:56:25.334514 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.334479 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:56:25.435430 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.435398 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-credential-socket\") pod \"e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.435640 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.435443 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-workload-certs\") pod \"e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.435640 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.435498 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d10edbd0-e449-459f-b955-6950c7142712-istio-token\") pod \"e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.435640 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.435525 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d10edbd0-e449-459f-b955-6950c7142712-istio-podinfo\") pod \"e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.435640 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.435563 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-istio-data\") pod \"e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.435640 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.435598 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-workload-socket\") pod \"e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.435640 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.435635 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d10edbd0-e449-459f-b955-6950c7142712-istiod-ca-cert\") pod \"e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.435867 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.435670 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24944\" (UniqueName: \"kubernetes.io/projected/d10edbd0-e449-459f-b955-6950c7142712-kube-api-access-24944\") pod \"e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.435867 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.435698 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-istio-envoy\") pod \"e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.536835 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.536802 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d10edbd0-e449-459f-b955-6950c7142712-istiod-ca-cert\") pod \"e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.536993 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.536872 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24944\" (UniqueName: \"kubernetes.io/projected/d10edbd0-e449-459f-b955-6950c7142712-kube-api-access-24944\") pod \"e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.536993 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.536913 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-istio-envoy\") pod \"e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.536993 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.536953 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-credential-socket\") pod \"e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.537158 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.536992 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-workload-certs\") pod \"e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.537158 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.537046 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d10edbd0-e449-459f-b955-6950c7142712-istio-token\") pod \"e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.537158 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.537076 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d10edbd0-e449-459f-b955-6950c7142712-istio-podinfo\") pod \"e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.537158 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.537124 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-istio-data\") pod \"e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.537353 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.537159 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-workload-socket\") pod \"e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.537413 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.537371 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-workload-certs\") pod \"e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.537491 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.537423 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-credential-socket\") pod \"e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.537555 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.537499 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-workload-socket\") pod \"e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.537555 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.537533 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d10edbd0-e449-459f-b955-6950c7142712-istiod-ca-cert\") pod \"e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.537730 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.537705 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-istio-data\") pod \"e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.539351 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.539327 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-istio-envoy\") pod \"e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.539802 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.539782 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d10edbd0-e449-459f-b955-6950c7142712-istio-podinfo\") pod \"e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.548535 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.548512 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d10edbd0-e449-459f-b955-6950c7142712-istio-token\") pod \"e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.548933 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.548900 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24944\" (UniqueName: \"kubernetes.io/projected/d10edbd0-e449-459f-b955-6950c7142712-kube-api-access-24944\") pod \"e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.616861 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.616832 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:25.667020 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.666983 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:56:25.771855 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.771831 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc"] Jul 24 03:56:25.773891 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:56:25.773863 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10edbd0_e449_459f_b955_6950c7142712.slice/crio-25d22a05da2bbc420b83ef4871a70432dbbacfec2bd1f25729f028417770ff2f WatchSource:0}: Error finding container 25d22a05da2bbc420b83ef4871a70432dbbacfec2bd1f25729f028417770ff2f: Status 404 returned error can't find the container with id 25d22a05da2bbc420b83ef4871a70432dbbacfec2bd1f25729f028417770ff2f Jul 24 03:56:25.776474 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.776432 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:56:25.776599 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.776518 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:56:25.776599 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:25.776551 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:56:26.284208 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:26.284163 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" event={"ID":"d10edbd0-e449-459f-b955-6950c7142712","Type":"ContainerStarted","Data":"57111512b1d3e1207f98a605f789c3ce0c0add55bb256c1231894e9527a2b092"} Jul 24 03:56:26.284383 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:26.284280 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" event={"ID":"d10edbd0-e449-459f-b955-6950c7142712","Type":"ContainerStarted","Data":"25d22a05da2bbc420b83ef4871a70432dbbacfec2bd1f25729f028417770ff2f"} Jul 24 03:56:26.306408 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:26.306358 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" podStartSLOduration=1.306340826 podStartE2EDuration="1.306340826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:56:26.303509857 +0000 UTC m=+2335.742929314" watchObservedRunningTime="2026-07-24 03:56:26.306340826 +0000 UTC m=+2335.745760286" Jul 24 03:56:26.617120 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:26.617085 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:26.622290 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:26.622263 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:27.293636 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:27.293601 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:27.298179 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:27.298148 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:28.244660 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:28.244612 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:56:28.280763 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:28.280731 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq" Jul 24 03:56:28.339155 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:28.339124 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:56:28.438901 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:28.438846 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:56:28.761604 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:28.761566 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:56:28.951747 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:28.951709 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:56:29.053480 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:29.053385 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:56:32.759244 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:32.759200 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:56:34.648474 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:34.647833 2579 patch_prober.go:28] interesting pod/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": read tcp 10.134.0.2:35820->10.134.0.62:15021: read: connection reset by peer" start-of-body= Jul 24 03:56:34.648474 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:34.647913 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" podUID="32705bc1-1bbd-43fe-a689-611ea28a7e69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": read tcp 10.134.0.2:35820->10.134.0.62:15021: read: connection reset by peer" Jul 24 03:56:35.343305 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:35.343270 2579 generic.go:356] "Generic (PLEG): container finished" podID="32705bc1-1bbd-43fe-a689-611ea28a7e69" containerID="0548f54b379d1b64e5548fca65b4b2ca0bf3b020e22a578461c3e67975b226a4" exitCode=0 Jul 24 03:56:35.343506 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:35.343343 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" event={"ID":"32705bc1-1bbd-43fe-a689-611ea28a7e69","Type":"ContainerDied","Data":"0548f54b379d1b64e5548fca65b4b2ca0bf3b020e22a578461c3e67975b226a4"} Jul 24 03:56:35.343830 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:35.343814 2579 scope.go:117] "RemoveContainer" containerID="0548f54b379d1b64e5548fca65b4b2ca0bf3b020e22a578461c3e67975b226a4" Jul 24 03:56:35.344943 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:35.344915 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:56:35.345001 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:35.344985 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:56:35.345045 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:35.345021 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:56:36.351038 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:36.350995 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" event={"ID":"32705bc1-1bbd-43fe-a689-611ea28a7e69","Type":"ContainerStarted","Data":"8065c958ddc55ec7d386a6b810e23d0345d2d201c4285eb573525c85aa562bfa"} Jul 24 03:56:37.172894 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:37.172862 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:37.173162 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:37.173120 2579 patch_prober.go:28] interesting pod/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 24 03:56:37.173318 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:37.173293 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" podUID="32705bc1-1bbd-43fe-a689-611ea28a7e69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 24 03:56:38.186689 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:38.186632 2579 patch_prober.go:28] interesting pod/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 24 03:56:38.187215 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:38.186771 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" podUID="32705bc1-1bbd-43fe-a689-611ea28a7e69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 24 03:56:39.173203 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:39.173144 2579 patch_prober.go:28] interesting pod/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 24 03:56:39.173422 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:39.173236 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" podUID="32705bc1-1bbd-43fe-a689-611ea28a7e69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 24 03:56:40.172832 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:40.172791 2579 patch_prober.go:28] interesting pod/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 24 03:56:40.173481 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:40.173419 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" podUID="32705bc1-1bbd-43fe-a689-611ea28a7e69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 24 03:56:41.176735 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:41.176706 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:41.177183 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:41.177018 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:41.177810 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:41.177789 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:56:41.381439 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:41.381406 2579 generic.go:356] "Generic (PLEG): container finished" podID="d10edbd0-e449-459f-b955-6950c7142712" containerID="57111512b1d3e1207f98a605f789c3ce0c0add55bb256c1231894e9527a2b092" exitCode=0 Jul 24 03:56:41.381623 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:41.381491 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" event={"ID":"d10edbd0-e449-459f-b955-6950c7142712","Type":"ContainerDied","Data":"57111512b1d3e1207f98a605f789c3ce0c0add55bb256c1231894e9527a2b092"} Jul 24 03:56:41.382132 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:41.382115 2579 scope.go:117] "RemoveContainer" containerID="57111512b1d3e1207f98a605f789c3ce0c0add55bb256c1231894e9527a2b092" Jul 24 03:56:41.383350 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:41.383321 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:56:41.383455 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:41.383387 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:56:41.383455 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:41.383423 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:56:42.293352 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:42.293309 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:42.398382 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:42.398335 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" event={"ID":"d10edbd0-e449-459f-b955-6950c7142712","Type":"ContainerStarted","Data":"48748d119320bca2322b1e6c69b16c2d65a9e7bb51759a8b554fda360923c9ad"} Jul 24 03:56:42.617625 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:42.617581 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:42.618818 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:42.618790 2579 patch_prober.go:28] interesting pod/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 24 03:56:42.618958 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:42.618852 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" podUID="d10edbd0-e449-459f-b955-6950c7142712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 24 03:56:43.617950 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:43.617905 2579 patch_prober.go:28] interesting pod/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 24 03:56:43.618370 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:43.617972 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" podUID="d10edbd0-e449-459f-b955-6950c7142712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 24 03:56:44.618259 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:44.618224 2579 patch_prober.go:28] interesting pod/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 24 03:56:44.618668 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:44.618281 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" podUID="d10edbd0-e449-459f-b955-6950c7142712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 24 03:56:45.618143 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:45.618105 2579 patch_prober.go:28] interesting pod/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 24 03:56:45.618337 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:45.618162 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" podUID="d10edbd0-e449-459f-b955-6950c7142712" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 24 03:56:46.621219 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:46.621189 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:46.621718 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:46.621701 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:46.622394 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:46.622377 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:56:48.270819 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:56:48.270775 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:57:07.771347 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:57:07.771314 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:57:48.657918 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:57:48.657882 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:58:24.035118 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:24.035026 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:58:24.947180 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:24.947151 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:58:35.569610 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:35.569564 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq"] Jul 24 03:58:35.570102 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:35.569886 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq" podUID="e2bdd23d-e644-4b6a-a347-f74eae14047a" containerName="maas-api" containerID="cri-o://e818121e6420e183962ee3dd8ae571cd472dd5f6492982c82bd3719f952c892e" gracePeriod=30 Jul 24 03:58:35.600835 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:35.600808 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:58:35.702808 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:35.702767 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:58:35.804051 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:35.803825 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:58:35.849637 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:35.849616 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq" Jul 24 03:58:35.865738 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:35.865713 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8zmh\" (UniqueName: \"kubernetes.io/projected/e2bdd23d-e644-4b6a-a347-f74eae14047a-kube-api-access-j8zmh\") pod \"e2bdd23d-e644-4b6a-a347-f74eae14047a\" (UID: \"e2bdd23d-e644-4b6a-a347-f74eae14047a\") " Jul 24 03:58:35.865832 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:35.865765 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2bdd23d-e644-4b6a-a347-f74eae14047a-maas-api-tls\") pod \"e2bdd23d-e644-4b6a-a347-f74eae14047a\" (UID: \"e2bdd23d-e644-4b6a-a347-f74eae14047a\") " Jul 24 03:58:35.865832 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:35.865807 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2bdd23d-e644-4b6a-a347-f74eae14047a-openshift-service-ca\") pod \"e2bdd23d-e644-4b6a-a347-f74eae14047a\" (UID: \"e2bdd23d-e644-4b6a-a347-f74eae14047a\") " Jul 24 03:58:35.866193 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:35.866168 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2bdd23d-e644-4b6a-a347-f74eae14047a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e2bdd23d-e644-4b6a-a347-f74eae14047a" (UID: "e2bdd23d-e644-4b6a-a347-f74eae14047a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 03:58:35.868316 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:35.868289 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2bdd23d-e644-4b6a-a347-f74eae14047a-kube-api-access-j8zmh" (OuterVolumeSpecName: "kube-api-access-j8zmh") pod "e2bdd23d-e644-4b6a-a347-f74eae14047a" (UID: "e2bdd23d-e644-4b6a-a347-f74eae14047a"). InnerVolumeSpecName "kube-api-access-j8zmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:58:35.868626 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:35.868605 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2bdd23d-e644-4b6a-a347-f74eae14047a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e2bdd23d-e644-4b6a-a347-f74eae14047a" (UID: "e2bdd23d-e644-4b6a-a347-f74eae14047a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 03:58:35.967337 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:35.967304 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8zmh\" (UniqueName: \"kubernetes.io/projected/e2bdd23d-e644-4b6a-a347-f74eae14047a-kube-api-access-j8zmh\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:58:35.967337 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:35.967336 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2bdd23d-e644-4b6a-a347-f74eae14047a-maas-api-tls\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:58:35.967545 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:35.967346 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2bdd23d-e644-4b6a-a347-f74eae14047a-openshift-service-ca\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:58:35.995724 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:35.995688 2579 generic.go:356] "Generic (PLEG): container finished" podID="e2bdd23d-e644-4b6a-a347-f74eae14047a" containerID="e818121e6420e183962ee3dd8ae571cd472dd5f6492982c82bd3719f952c892e" exitCode=0 Jul 24 03:58:35.995896 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:35.995733 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq" Jul 24 03:58:35.995896 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:35.995755 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq" event={"ID":"e2bdd23d-e644-4b6a-a347-f74eae14047a","Type":"ContainerDied","Data":"e818121e6420e183962ee3dd8ae571cd472dd5f6492982c82bd3719f952c892e"} Jul 24 03:58:35.995896 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:35.995801 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq" event={"ID":"e2bdd23d-e644-4b6a-a347-f74eae14047a","Type":"ContainerDied","Data":"99933aceb01abde773f8d87f12aa02c76de4578eb0d915469cbb07c51cbd3aaa"} Jul 24 03:58:35.995896 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:35.995823 2579 scope.go:117] "RemoveContainer" containerID="e818121e6420e183962ee3dd8ae571cd472dd5f6492982c82bd3719f952c892e" Jul 24 03:58:36.006289 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:36.006273 2579 scope.go:117] "RemoveContainer" containerID="e818121e6420e183962ee3dd8ae571cd472dd5f6492982c82bd3719f952c892e" Jul 24 03:58:36.006570 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:58:36.006551 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e818121e6420e183962ee3dd8ae571cd472dd5f6492982c82bd3719f952c892e\": container with ID starting with e818121e6420e183962ee3dd8ae571cd472dd5f6492982c82bd3719f952c892e not found: ID does not exist" containerID="e818121e6420e183962ee3dd8ae571cd472dd5f6492982c82bd3719f952c892e" Jul 24 03:58:36.006631 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:36.006580 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e818121e6420e183962ee3dd8ae571cd472dd5f6492982c82bd3719f952c892e"} err="failed to get container status \"e818121e6420e183962ee3dd8ae571cd472dd5f6492982c82bd3719f952c892e\": rpc error: code = NotFound desc = could not find container \"e818121e6420e183962ee3dd8ae571cd472dd5f6492982c82bd3719f952c892e\": container with ID starting with e818121e6420e183962ee3dd8ae571cd472dd5f6492982c82bd3719f952c892e not found: ID does not exist" Jul 24 03:58:36.019162 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:36.019138 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq"] Jul 24 03:58:36.023566 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:36.023547 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-93dff3-85c8cdf68c-hqbrq"] Jul 24 03:58:37.033268 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:37.033228 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2bdd23d-e644-4b6a-a347-f74eae14047a" path="/var/lib/kubelet/pods/e2bdd23d-e644-4b6a-a347-f74eae14047a/volumes" Jul 24 03:58:40.151529 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:40.151490 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:58:40.564736 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:40.564640 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:58:46.401059 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:46.401025 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:58:46.654247 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:46.654153 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w"] Jul 24 03:58:46.654517 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:46.654421 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" podUID="32705bc1-1bbd-43fe-a689-611ea28a7e69" containerName="istio-proxy" containerID="cri-o://8065c958ddc55ec7d386a6b810e23d0345d2d201c4285eb573525c85aa562bfa" gracePeriod=30 Jul 24 03:58:47.526565 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:47.526494 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:58:51.900485 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:51.900437 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:58:52.021211 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.021118 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-istio-envoy\") pod \"32705bc1-1bbd-43fe-a689-611ea28a7e69\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " Jul 24 03:58:52.021211 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.021158 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32705bc1-1bbd-43fe-a689-611ea28a7e69-istio-token\") pod \"32705bc1-1bbd-43fe-a689-611ea28a7e69\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " Jul 24 03:58:52.021211 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.021199 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-credential-socket\") pod \"32705bc1-1bbd-43fe-a689-611ea28a7e69\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " Jul 24 03:58:52.021541 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.021220 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-workload-certs\") pod \"32705bc1-1bbd-43fe-a689-611ea28a7e69\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " Jul 24 03:58:52.021541 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.021288 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jw9c8\" (UniqueName: \"kubernetes.io/projected/32705bc1-1bbd-43fe-a689-611ea28a7e69-kube-api-access-jw9c8\") pod \"32705bc1-1bbd-43fe-a689-611ea28a7e69\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " Jul 24 03:58:52.021541 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.021312 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-workload-socket\") pod \"32705bc1-1bbd-43fe-a689-611ea28a7e69\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " Jul 24 03:58:52.021541 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.021388 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32705bc1-1bbd-43fe-a689-611ea28a7e69-istio-podinfo\") pod \"32705bc1-1bbd-43fe-a689-611ea28a7e69\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " Jul 24 03:58:52.021541 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.021432 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32705bc1-1bbd-43fe-a689-611ea28a7e69-istiod-ca-cert\") pod \"32705bc1-1bbd-43fe-a689-611ea28a7e69\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " Jul 24 03:58:52.021541 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.021505 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-istio-data\") pod \"32705bc1-1bbd-43fe-a689-611ea28a7e69\" (UID: \"32705bc1-1bbd-43fe-a689-611ea28a7e69\") " Jul 24 03:58:52.021855 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.021601 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "32705bc1-1bbd-43fe-a689-611ea28a7e69" (UID: "32705bc1-1bbd-43fe-a689-611ea28a7e69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:58:52.021855 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.021616 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "32705bc1-1bbd-43fe-a689-611ea28a7e69" (UID: "32705bc1-1bbd-43fe-a689-611ea28a7e69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:58:52.022007 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.021896 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-credential-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:58:52.022007 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.021915 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-workload-certs\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:58:52.022007 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.021916 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "32705bc1-1bbd-43fe-a689-611ea28a7e69" (UID: "32705bc1-1bbd-43fe-a689-611ea28a7e69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:58:52.022007 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.021959 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32705bc1-1bbd-43fe-a689-611ea28a7e69-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "32705bc1-1bbd-43fe-a689-611ea28a7e69" (UID: "32705bc1-1bbd-43fe-a689-611ea28a7e69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 03:58:52.022728 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.022681 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-istio-data" (OuterVolumeSpecName: "istio-data") pod "32705bc1-1bbd-43fe-a689-611ea28a7e69" (UID: "32705bc1-1bbd-43fe-a689-611ea28a7e69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:58:52.023729 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.023702 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "32705bc1-1bbd-43fe-a689-611ea28a7e69" (UID: "32705bc1-1bbd-43fe-a689-611ea28a7e69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:58:52.023729 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.023715 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32705bc1-1bbd-43fe-a689-611ea28a7e69-istio-token" (OuterVolumeSpecName: "istio-token") pod "32705bc1-1bbd-43fe-a689-611ea28a7e69" (UID: "32705bc1-1bbd-43fe-a689-611ea28a7e69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:58:52.023874 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.023748 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32705bc1-1bbd-43fe-a689-611ea28a7e69-kube-api-access-jw9c8" (OuterVolumeSpecName: "kube-api-access-jw9c8") pod "32705bc1-1bbd-43fe-a689-611ea28a7e69" (UID: "32705bc1-1bbd-43fe-a689-611ea28a7e69"). InnerVolumeSpecName "kube-api-access-jw9c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:58:52.023913 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.023876 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32705bc1-1bbd-43fe-a689-611ea28a7e69-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "32705bc1-1bbd-43fe-a689-611ea28a7e69" (UID: "32705bc1-1bbd-43fe-a689-611ea28a7e69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 03:58:52.082053 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.082024 2579 generic.go:356] "Generic (PLEG): container finished" podID="32705bc1-1bbd-43fe-a689-611ea28a7e69" containerID="8065c958ddc55ec7d386a6b810e23d0345d2d201c4285eb573525c85aa562bfa" exitCode=0 Jul 24 03:58:52.082179 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.082071 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" Jul 24 03:58:52.082179 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.082101 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" event={"ID":"32705bc1-1bbd-43fe-a689-611ea28a7e69","Type":"ContainerDied","Data":"8065c958ddc55ec7d386a6b810e23d0345d2d201c4285eb573525c85aa562bfa"} Jul 24 03:58:52.082179 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.082152 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w" event={"ID":"32705bc1-1bbd-43fe-a689-611ea28a7e69","Type":"ContainerDied","Data":"91d58ce680f3999c401b15f61fe5f0d18b1d125ecc65ac9ca0bb5a0bcfea88cf"} Jul 24 03:58:52.082179 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.082175 2579 scope.go:117] "RemoveContainer" containerID="8065c958ddc55ec7d386a6b810e23d0345d2d201c4285eb573525c85aa562bfa" Jul 24 03:58:52.091530 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.091512 2579 scope.go:117] "RemoveContainer" containerID="0548f54b379d1b64e5548fca65b4b2ca0bf3b020e22a578461c3e67975b226a4" Jul 24 03:58:52.099618 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.099598 2579 scope.go:117] "RemoveContainer" containerID="8065c958ddc55ec7d386a6b810e23d0345d2d201c4285eb573525c85aa562bfa" Jul 24 03:58:52.099909 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:58:52.099890 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8065c958ddc55ec7d386a6b810e23d0345d2d201c4285eb573525c85aa562bfa\": container with ID starting with 8065c958ddc55ec7d386a6b810e23d0345d2d201c4285eb573525c85aa562bfa not found: ID does not exist" containerID="8065c958ddc55ec7d386a6b810e23d0345d2d201c4285eb573525c85aa562bfa" Jul 24 03:58:52.099959 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.099917 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8065c958ddc55ec7d386a6b810e23d0345d2d201c4285eb573525c85aa562bfa"} err="failed to get container status \"8065c958ddc55ec7d386a6b810e23d0345d2d201c4285eb573525c85aa562bfa\": rpc error: code = NotFound desc = could not find container \"8065c958ddc55ec7d386a6b810e23d0345d2d201c4285eb573525c85aa562bfa\": container with ID starting with 8065c958ddc55ec7d386a6b810e23d0345d2d201c4285eb573525c85aa562bfa not found: ID does not exist" Jul 24 03:58:52.099959 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.099933 2579 scope.go:117] "RemoveContainer" containerID="0548f54b379d1b64e5548fca65b4b2ca0bf3b020e22a578461c3e67975b226a4" Jul 24 03:58:52.100135 ip-10-0-134-147 kubenswrapper[2579]: E0724 03:58:52.100120 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0548f54b379d1b64e5548fca65b4b2ca0bf3b020e22a578461c3e67975b226a4\": container with ID starting with 0548f54b379d1b64e5548fca65b4b2ca0bf3b020e22a578461c3e67975b226a4 not found: ID does not exist" containerID="0548f54b379d1b64e5548fca65b4b2ca0bf3b020e22a578461c3e67975b226a4" Jul 24 03:58:52.100175 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.100141 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0548f54b379d1b64e5548fca65b4b2ca0bf3b020e22a578461c3e67975b226a4"} err="failed to get container status \"0548f54b379d1b64e5548fca65b4b2ca0bf3b020e22a578461c3e67975b226a4\": rpc error: code = NotFound desc = could not find container \"0548f54b379d1b64e5548fca65b4b2ca0bf3b020e22a578461c3e67975b226a4\": container with ID starting with 0548f54b379d1b64e5548fca65b4b2ca0bf3b020e22a578461c3e67975b226a4 not found: ID does not exist" Jul 24 03:58:52.109215 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.105691 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w"] Jul 24 03:58:52.113514 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.113492 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-93dff3-openshift-default-8454798c5b-fmb2w"] Jul 24 03:58:52.123035 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.123016 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw9c8\" (UniqueName: \"kubernetes.io/projected/32705bc1-1bbd-43fe-a689-611ea28a7e69-kube-api-access-jw9c8\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:58:52.123121 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.123036 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-workload-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:58:52.123121 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.123046 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32705bc1-1bbd-43fe-a689-611ea28a7e69-istio-podinfo\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:58:52.123121 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.123055 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32705bc1-1bbd-43fe-a689-611ea28a7e69-istiod-ca-cert\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:58:52.123121 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.123064 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-istio-data\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:58:52.123121 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.123071 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32705bc1-1bbd-43fe-a689-611ea28a7e69-istio-envoy\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:58:52.123121 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:52.123078 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32705bc1-1bbd-43fe-a689-611ea28a7e69-istio-token\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:58:53.033909 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:53.033873 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32705bc1-1bbd-43fe-a689-611ea28a7e69" path="/var/lib/kubelet/pods/32705bc1-1bbd-43fe-a689-611ea28a7e69/volumes" Jul 24 03:58:57.464760 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:57.464732 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:58:57.540951 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:57.540915 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:58:57.606121 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:58:57.606087 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:02.128845 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:02.128801 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:02.523217 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:02.523132 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:08.690757 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:08.690718 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:08.794828 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:08.794796 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:08.947881 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:08.947800 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc"] Jul 24 03:59:08.948074 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:08.948050 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" podUID="d10edbd0-e449-459f-b955-6950c7142712" containerName="istio-proxy" containerID="cri-o://48748d119320bca2322b1e6c69b16c2d65a9e7bb51759a8b554fda360923c9ad" gracePeriod=30 Jul 24 03:59:08.964087 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:08.964063 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:09.198359 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:09.198277 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:09.532911 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:09.532835 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:09.830020 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:09.829926 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl"] Jul 24 03:59:09.830657 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:09.830630 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32705bc1-1bbd-43fe-a689-611ea28a7e69" containerName="istio-proxy" Jul 24 03:59:09.830657 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:09.830656 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="32705bc1-1bbd-43fe-a689-611ea28a7e69" containerName="istio-proxy" Jul 24 03:59:09.830867 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:09.830690 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2bdd23d-e644-4b6a-a347-f74eae14047a" containerName="maas-api" Jul 24 03:59:09.830867 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:09.830700 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2bdd23d-e644-4b6a-a347-f74eae14047a" containerName="maas-api" Jul 24 03:59:09.830867 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:09.830798 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2bdd23d-e644-4b6a-a347-f74eae14047a" containerName="maas-api" Jul 24 03:59:09.830867 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:09.830812 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="32705bc1-1bbd-43fe-a689-611ea28a7e69" containerName="istio-proxy" Jul 24 03:59:09.830867 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:09.830820 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="32705bc1-1bbd-43fe-a689-611ea28a7e69" containerName="istio-proxy" Jul 24 03:59:09.834404 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:09.834384 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:09.837097 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:09.837075 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f822cd-openshift-default-dockercfg-wrv9j\"" Jul 24 03:59:09.848068 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:09.848041 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl"] Jul 24 03:59:09.862638 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:09.862611 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:09.997277 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:09.997249 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-istio-data\") pod \"e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:09.997447 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:09.997295 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-workload-socket\") pod \"e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:09.997447 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:09.997343 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-workload-certs\") pod \"e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:09.997447 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:09.997360 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-credential-socket\") pod \"e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:09.997569 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:09.997453 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ede84b0-7533-415a-b73e-31454c67c06a-istio-podinfo\") pod \"e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:09.997569 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:09.997513 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ede84b0-7533-415a-b73e-31454c67c06a-istio-token\") pod \"e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:09.997639 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:09.997567 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxjpw\" (UniqueName: \"kubernetes.io/projected/7ede84b0-7533-415a-b73e-31454c67c06a-kube-api-access-pxjpw\") pod \"e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:09.997639 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:09.997598 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ede84b0-7533-415a-b73e-31454c67c06a-istiod-ca-cert\") pod \"e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:09.997639 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:09.997632 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-istio-envoy\") pod \"e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:10.098787 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:10.098706 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-istio-data\") pod \"e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:10.098787 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:10.098750 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-workload-socket\") pod \"e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:10.098787 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:10.098775 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-workload-certs\") pod \"e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:10.099079 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:10.098792 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-credential-socket\") pod \"e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:10.099079 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:10.098821 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ede84b0-7533-415a-b73e-31454c67c06a-istio-podinfo\") pod \"e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:10.099079 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:10.098843 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ede84b0-7533-415a-b73e-31454c67c06a-istio-token\") pod \"e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:10.099079 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:10.098883 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxjpw\" (UniqueName: \"kubernetes.io/projected/7ede84b0-7533-415a-b73e-31454c67c06a-kube-api-access-pxjpw\") pod \"e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:10.099079 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:10.098923 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ede84b0-7533-415a-b73e-31454c67c06a-istiod-ca-cert\") pod \"e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:10.099079 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:10.098978 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-istio-envoy\") pod \"e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:10.099411 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:10.099147 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-istio-data\") pod \"e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:10.099411 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:10.099201 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-workload-socket\") pod \"e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:10.099411 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:10.099242 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-credential-socket\") pod \"e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:10.099411 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:10.099274 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-workload-certs\") pod \"e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:10.099838 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:10.099816 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ede84b0-7533-415a-b73e-31454c67c06a-istiod-ca-cert\") pod \"e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:10.101685 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:10.101662 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-istio-envoy\") pod \"e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:10.101899 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:10.101874 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ede84b0-7533-415a-b73e-31454c67c06a-istio-podinfo\") pod \"e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:10.109359 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:10.109331 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ede84b0-7533-415a-b73e-31454c67c06a-istio-token\") pod \"e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:10.109703 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:10.109669 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxjpw\" (UniqueName: \"kubernetes.io/projected/7ede84b0-7533-415a-b73e-31454c67c06a-kube-api-access-pxjpw\") pod \"e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:10.145388 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:10.145358 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:10.298536 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:10.298511 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl"] Jul 24 03:59:10.300512 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:59:10.300483 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ede84b0_7533_415a_b73e_31454c67c06a.slice/crio-2b5530bdc782f5fb2a44d019222cb0d05d250f48596a5b483269530bcb03f392 WatchSource:0}: Error finding container 2b5530bdc782f5fb2a44d019222cb0d05d250f48596a5b483269530bcb03f392: Status 404 returned error can't find the container with id 2b5530bdc782f5fb2a44d019222cb0d05d250f48596a5b483269530bcb03f392 Jul 24 03:59:10.302650 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:10.302616 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:59:10.302726 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:10.302701 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:59:10.302764 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:10.302732 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:59:10.391353 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:10.391330 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:11.182941 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:11.182902 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" event={"ID":"7ede84b0-7533-415a-b73e-31454c67c06a","Type":"ContainerStarted","Data":"bdf902c539d94c5cf37b4e6dadd862f203dcf158048f87e0fe0b98044ad620de"} Jul 24 03:59:11.183319 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:11.182963 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" event={"ID":"7ede84b0-7533-415a-b73e-31454c67c06a","Type":"ContainerStarted","Data":"2b5530bdc782f5fb2a44d019222cb0d05d250f48596a5b483269530bcb03f392"} Jul 24 03:59:11.205260 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:11.205214 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" podStartSLOduration=2.205199087 podStartE2EDuration="2.205199087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:59:11.203487545 +0000 UTC m=+2500.642907008" watchObservedRunningTime="2026-07-24 03:59:11.205199087 +0000 UTC m=+2500.644618547" Jul 24 03:59:12.145726 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:12.145684 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:12.150754 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:12.150726 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:12.188174 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:12.188140 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:12.189025 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:12.189007 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:13.137576 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:13.137541 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:13.238512 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:13.238485 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:13.384584 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:13.384548 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:13.591229 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:13.591142 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:13.892510 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:13.892482 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:13.991030 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:13.991004 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:14.200980 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.200948 2579 generic.go:356] "Generic (PLEG): container finished" podID="d10edbd0-e449-459f-b955-6950c7142712" containerID="48748d119320bca2322b1e6c69b16c2d65a9e7bb51759a8b554fda360923c9ad" exitCode=0 Jul 24 03:59:14.201133 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.201022 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" event={"ID":"d10edbd0-e449-459f-b955-6950c7142712","Type":"ContainerDied","Data":"48748d119320bca2322b1e6c69b16c2d65a9e7bb51759a8b554fda360923c9ad"} Jul 24 03:59:14.201133 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.201079 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" event={"ID":"d10edbd0-e449-459f-b955-6950c7142712","Type":"ContainerDied","Data":"25d22a05da2bbc420b83ef4871a70432dbbacfec2bd1f25729f028417770ff2f"} Jul 24 03:59:14.201133 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.201097 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25d22a05da2bbc420b83ef4871a70432dbbacfec2bd1f25729f028417770ff2f" Jul 24 03:59:14.201133 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.201112 2579 scope.go:117] "RemoveContainer" containerID="57111512b1d3e1207f98a605f789c3ce0c0add55bb256c1231894e9527a2b092" Jul 24 03:59:14.209024 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.209001 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:59:14.242219 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.242194 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d10edbd0-e449-459f-b955-6950c7142712-istio-token\") pod \"d10edbd0-e449-459f-b955-6950c7142712\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " Jul 24 03:59:14.242667 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.242230 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-istio-data\") pod \"d10edbd0-e449-459f-b955-6950c7142712\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " Jul 24 03:59:14.242667 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.242279 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-workload-certs\") pod \"d10edbd0-e449-459f-b955-6950c7142712\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " Jul 24 03:59:14.242667 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.242324 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-workload-socket\") pod \"d10edbd0-e449-459f-b955-6950c7142712\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " Jul 24 03:59:14.242667 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.242390 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24944\" (UniqueName: \"kubernetes.io/projected/d10edbd0-e449-459f-b955-6950c7142712-kube-api-access-24944\") pod \"d10edbd0-e449-459f-b955-6950c7142712\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " Jul 24 03:59:14.242667 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.242418 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-credential-socket\") pod \"d10edbd0-e449-459f-b955-6950c7142712\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " Jul 24 03:59:14.242667 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.242495 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d10edbd0-e449-459f-b955-6950c7142712-istio-podinfo\") pod \"d10edbd0-e449-459f-b955-6950c7142712\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " Jul 24 03:59:14.242667 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.242531 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-istio-envoy\") pod \"d10edbd0-e449-459f-b955-6950c7142712\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " Jul 24 03:59:14.242667 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.242561 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d10edbd0-e449-459f-b955-6950c7142712-istiod-ca-cert\") pod \"d10edbd0-e449-459f-b955-6950c7142712\" (UID: \"d10edbd0-e449-459f-b955-6950c7142712\") " Jul 24 03:59:14.244026 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.243163 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d10edbd0-e449-459f-b955-6950c7142712" (UID: "d10edbd0-e449-459f-b955-6950c7142712"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:59:14.244026 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.243387 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d10edbd0-e449-459f-b955-6950c7142712" (UID: "d10edbd0-e449-459f-b955-6950c7142712"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:59:14.244026 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.243414 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d10edbd0-e449-459f-b955-6950c7142712" (UID: "d10edbd0-e449-459f-b955-6950c7142712"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:59:14.244026 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.243701 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-istio-data" (OuterVolumeSpecName: "istio-data") pod "d10edbd0-e449-459f-b955-6950c7142712" (UID: "d10edbd0-e449-459f-b955-6950c7142712"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:59:14.244026 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.243742 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d10edbd0-e449-459f-b955-6950c7142712-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d10edbd0-e449-459f-b955-6950c7142712" (UID: "d10edbd0-e449-459f-b955-6950c7142712"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 03:59:14.245983 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.245952 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d10edbd0-e449-459f-b955-6950c7142712" (UID: "d10edbd0-e449-459f-b955-6950c7142712"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 03:59:14.246203 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.246178 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d10edbd0-e449-459f-b955-6950c7142712-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d10edbd0-e449-459f-b955-6950c7142712" (UID: "d10edbd0-e449-459f-b955-6950c7142712"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 03:59:14.246606 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.246587 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d10edbd0-e449-459f-b955-6950c7142712-istio-token" (OuterVolumeSpecName: "istio-token") pod "d10edbd0-e449-459f-b955-6950c7142712" (UID: "d10edbd0-e449-459f-b955-6950c7142712"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:59:14.246911 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.246871 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d10edbd0-e449-459f-b955-6950c7142712-kube-api-access-24944" (OuterVolumeSpecName: "kube-api-access-24944") pod "d10edbd0-e449-459f-b955-6950c7142712" (UID: "d10edbd0-e449-459f-b955-6950c7142712"). InnerVolumeSpecName "kube-api-access-24944". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 03:59:14.344296 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.344263 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24944\" (UniqueName: \"kubernetes.io/projected/d10edbd0-e449-459f-b955-6950c7142712-kube-api-access-24944\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:59:14.344296 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.344291 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-credential-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:59:14.344296 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.344301 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d10edbd0-e449-459f-b955-6950c7142712-istio-podinfo\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:59:14.344571 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.344309 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-istio-envoy\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:59:14.344571 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.344317 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d10edbd0-e449-459f-b955-6950c7142712-istiod-ca-cert\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:59:14.344571 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.344325 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d10edbd0-e449-459f-b955-6950c7142712-istio-token\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:59:14.344571 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.344333 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-istio-data\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:59:14.344571 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.344341 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-workload-certs\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:59:14.344571 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:14.344348 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d10edbd0-e449-459f-b955-6950c7142712-workload-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 03:59:15.208582 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:15.208536 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc" Jul 24 03:59:15.244412 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:15.244381 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc"] Jul 24 03:59:15.247053 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:15.247025 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b0d347-openshift-default-794bfbcd88-ccmhc"] Jul 24 03:59:17.033995 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:17.033964 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d10edbd0-e449-459f-b955-6950c7142712" path="/var/lib/kubelet/pods/d10edbd0-e449-459f-b955-6950c7142712/volumes" Jul 24 03:59:17.662056 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:17.662023 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:18.136816 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.136783 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:18.431069 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.430989 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg"] Jul 24 03:59:18.431685 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.431668 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d10edbd0-e449-459f-b955-6950c7142712" containerName="istio-proxy" Jul 24 03:59:18.431733 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.431689 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d10edbd0-e449-459f-b955-6950c7142712" containerName="istio-proxy" Jul 24 03:59:18.431733 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.431726 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32705bc1-1bbd-43fe-a689-611ea28a7e69" containerName="istio-proxy" Jul 24 03:59:18.431818 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.431735 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="32705bc1-1bbd-43fe-a689-611ea28a7e69" containerName="istio-proxy" Jul 24 03:59:18.431860 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.431849 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d10edbd0-e449-459f-b955-6950c7142712" containerName="istio-proxy" Jul 24 03:59:18.437023 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.437002 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.439513 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.439487 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b23325-openshift-default-dockercfg-sc5mp\"" Jul 24 03:59:18.448370 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.448347 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg"] Jul 24 03:59:18.467380 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.467347 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:18.485073 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.485048 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istio-data\") pod \"e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.485193 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.485091 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-credential-socket\") pod \"e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.485193 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.485147 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istio-token\") pod \"e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.485271 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.485192 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-workload-socket\") pod \"e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.485316 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.485274 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-workload-certs\") pod \"e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.485316 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.485297 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx6j8\" (UniqueName: \"kubernetes.io/projected/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-kube-api-access-xx6j8\") pod \"e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.485407 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.485320 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istio-envoy\") pod \"e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.485407 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.485338 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istio-podinfo\") pod \"e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.485407 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.485361 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istiod-ca-cert\") pod \"e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.567594 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.567557 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:18.586478 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.586431 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istio-data\") pod \"e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.586657 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.586538 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-credential-socket\") pod \"e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.586657 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.586603 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istio-token\") pod \"e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.586657 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.586631 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-workload-socket\") pod \"e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.586828 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.586718 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-workload-certs\") pod \"e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.586828 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.586758 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx6j8\" (UniqueName: \"kubernetes.io/projected/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-kube-api-access-xx6j8\") pod \"e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.586828 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.586797 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istio-envoy\") pod \"e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.586828 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.586822 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istio-podinfo\") pod \"e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.587033 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.586847 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istiod-ca-cert\") pod \"e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.587033 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.586853 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istio-data\") pod \"e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.587174 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.587149 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-credential-socket\") pod \"e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.587698 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.587416 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-workload-certs\") pod \"e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.587897 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.587613 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-workload-socket\") pod \"e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.587998 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.587916 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istiod-ca-cert\") pod \"e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.589399 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.589376 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istio-envoy\") pod \"e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.589640 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.589606 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istio-podinfo\") pod \"e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.595986 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.595959 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istio-token\") pod \"e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.596211 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.596185 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx6j8\" (UniqueName: \"kubernetes.io/projected/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-kube-api-access-xx6j8\") pod \"e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.748425 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.748334 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:18.928580 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.928559 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg"] Jul 24 03:59:18.931362 ip-10-0-134-147 kubenswrapper[2579]: W0724 03:59:18.931338 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a9a9e2_5c88_46d6_afbe_c7deb58bb5bd.slice/crio-b54e360b0e581e20bc99f895d3ffc207f46fe3808575cdbf1046b714c5195206 WatchSource:0}: Error finding container b54e360b0e581e20bc99f895d3ffc207f46fe3808575cdbf1046b714c5195206: Status 404 returned error can't find the container with id b54e360b0e581e20bc99f895d3ffc207f46fe3808575cdbf1046b714c5195206 Jul 24 03:59:18.933845 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.933812 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:59:18.933954 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.933874 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:59:18.933954 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:18.933907 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:59:19.231690 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:19.231654 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" event={"ID":"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd","Type":"ContainerStarted","Data":"cf3bf87f9d033fb7de96ab8892416b07181e37f072f1c59af3b97554aaab42ed"} Jul 24 03:59:19.232070 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:19.231711 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" event={"ID":"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd","Type":"ContainerStarted","Data":"b54e360b0e581e20bc99f895d3ffc207f46fe3808575cdbf1046b714c5195206"} Jul 24 03:59:19.252751 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:19.252702 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" podStartSLOduration=1.252687299 podStartE2EDuration="1.252687299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 03:59:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 03:59:19.251692945 +0000 UTC m=+2508.691112405" watchObservedRunningTime="2026-07-24 03:59:19.252687299 +0000 UTC m=+2508.692106758" Jul 24 03:59:19.749361 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:19.749323 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:19.754185 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:19.754158 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:20.237866 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:20.237833 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:20.238851 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:20.238835 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:21.229602 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:21.229555 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:21.325429 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:21.325396 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:21.401641 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:21.401583 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:21.563910 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:21.563831 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:21.922009 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:21.921928 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:22.024227 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:22.024194 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:25.785884 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:25.785852 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:28.228704 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:28.228663 2579 patch_prober.go:28] interesting pod/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 03:59:28.229208 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:28.228734 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" podUID="7ede84b0-7533-415a-b73e-31454c67c06a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 03:59:29.291734 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:29.291689 2579 generic.go:356] "Generic (PLEG): container finished" podID="7ede84b0-7533-415a-b73e-31454c67c06a" containerID="bdf902c539d94c5cf37b4e6dadd862f203dcf158048f87e0fe0b98044ad620de" exitCode=0 Jul 24 03:59:29.292610 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:29.292320 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" event={"ID":"7ede84b0-7533-415a-b73e-31454c67c06a","Type":"ContainerDied","Data":"bdf902c539d94c5cf37b4e6dadd862f203dcf158048f87e0fe0b98044ad620de"} Jul 24 03:59:29.293288 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:29.293271 2579 scope.go:117] "RemoveContainer" containerID="bdf902c539d94c5cf37b4e6dadd862f203dcf158048f87e0fe0b98044ad620de" Jul 24 03:59:29.295702 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:29.295659 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:59:29.295907 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:29.295884 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:59:29.296090 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:29.296052 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:59:30.303610 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:30.303575 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" event={"ID":"7ede84b0-7533-415a-b73e-31454c67c06a","Type":"ContainerStarted","Data":"fc57c4adfbaf58cc1e3a1d9e4a5ecfa6a03275bc1ad1e482ba750ecbf8bb3303"} Jul 24 03:59:31.145789 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:31.145764 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:31.145962 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:31.145944 2579 patch_prober.go:28] interesting pod/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 24 03:59:31.146019 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:31.145990 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" podUID="7ede84b0-7533-415a-b73e-31454c67c06a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 24 03:59:32.145773 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:32.145737 2579 patch_prober.go:28] interesting pod/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 24 03:59:32.146176 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:32.145791 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" podUID="7ede84b0-7533-415a-b73e-31454c67c06a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 24 03:59:33.146495 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:33.146441 2579 patch_prober.go:28] interesting pod/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 24 03:59:33.146879 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:33.146516 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" podUID="7ede84b0-7533-415a-b73e-31454c67c06a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 24 03:59:34.146108 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:34.146065 2579 patch_prober.go:28] interesting pod/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 24 03:59:34.146278 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:34.146128 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" podUID="7ede84b0-7533-415a-b73e-31454c67c06a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 24 03:59:35.149528 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:35.149494 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:35.149988 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:35.149793 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:35.150455 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:35.150434 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 03:59:36.280603 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:36.280562 2579 patch_prober.go:28] interesting pod/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 03:59:36.280989 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:36.280614 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" podUID="41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 03:59:37.350467 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:37.350433 2579 generic.go:356] "Generic (PLEG): container finished" podID="41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd" containerID="cf3bf87f9d033fb7de96ab8892416b07181e37f072f1c59af3b97554aaab42ed" exitCode=0 Jul 24 03:59:37.351022 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:37.350509 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" event={"ID":"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd","Type":"ContainerDied","Data":"cf3bf87f9d033fb7de96ab8892416b07181e37f072f1c59af3b97554aaab42ed"} Jul 24 03:59:37.351104 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:37.351085 2579 scope.go:117] "RemoveContainer" containerID="cf3bf87f9d033fb7de96ab8892416b07181e37f072f1c59af3b97554aaab42ed" Jul 24 03:59:37.352231 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:37.352204 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:59:37.352305 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:37.352271 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:59:37.352349 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:37.352304 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 03:59:38.358097 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:38.358059 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" event={"ID":"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd","Type":"ContainerStarted","Data":"f9eed29caefddaa2747a84aade44652a7b73a02e9b3a71e5807edd79a781b7f6"} Jul 24 03:59:38.748716 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:38.748617 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:38.748864 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:38.748823 2579 patch_prober.go:28] interesting pod/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 24 03:59:38.748905 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:38.748864 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" podUID="41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 24 03:59:39.749352 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:39.749314 2579 patch_prober.go:28] interesting pod/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 24 03:59:39.749851 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:39.749370 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" podUID="41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 24 03:59:40.749244 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:40.749209 2579 patch_prober.go:28] interesting pod/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 24 03:59:40.749424 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:40.749262 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" podUID="41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 24 03:59:41.748986 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:41.748946 2579 patch_prober.go:28] interesting pod/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 24 03:59:41.749176 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:41.749012 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" podUID="41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 24 03:59:42.752567 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:42.752538 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:42.753037 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:42.752978 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:42.753975 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:42.753952 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 03:59:49.559536 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:49.559452 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:49.638852 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:49.638819 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:49.707407 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:49.707373 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:49.807247 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:49.807192 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:53.926114 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:53.926082 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 03:59:54.318248 ip-10-0-134-147 kubenswrapper[2579]: I0724 03:59:54.318129 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:00.326191 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:00.326150 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:00.426877 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:00.426841 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:00.591324 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:00.591244 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl"] Jul 24 04:00:00.591493 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:00.591449 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" podUID="7ede84b0-7533-415a-b73e-31454c67c06a" containerName="istio-proxy" containerID="cri-o://fc57c4adfbaf58cc1e3a1d9e4a5ecfa6a03275bc1ad1e482ba750ecbf8bb3303" gracePeriod=30 Jul 24 04:00:00.637206 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:00.637165 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:00.871563 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:00.871528 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:05.152553 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:05.152508 2579 patch_prober.go:28] interesting pod/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 24 04:00:05.152958 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:05.152605 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" podUID="7ede84b0-7533-415a-b73e-31454c67c06a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 04:00:05.839917 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:05.839895 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 04:00:05.970810 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:05.970776 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-istio-envoy\") pod \"7ede84b0-7533-415a-b73e-31454c67c06a\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " Jul 24 04:00:05.971010 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:05.970819 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-workload-certs\") pod \"7ede84b0-7533-415a-b73e-31454c67c06a\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " Jul 24 04:00:05.971010 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:05.970849 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-workload-socket\") pod \"7ede84b0-7533-415a-b73e-31454c67c06a\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " Jul 24 04:00:05.971010 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:05.970886 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-credential-socket\") pod \"7ede84b0-7533-415a-b73e-31454c67c06a\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " Jul 24 04:00:05.971010 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:05.970906 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ede84b0-7533-415a-b73e-31454c67c06a-istio-podinfo\") pod \"7ede84b0-7533-415a-b73e-31454c67c06a\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " Jul 24 04:00:05.971010 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:05.970958 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ede84b0-7533-415a-b73e-31454c67c06a-istio-token\") pod \"7ede84b0-7533-415a-b73e-31454c67c06a\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " Jul 24 04:00:05.971010 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:05.970999 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ede84b0-7533-415a-b73e-31454c67c06a-istiod-ca-cert\") pod \"7ede84b0-7533-415a-b73e-31454c67c06a\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " Jul 24 04:00:05.971329 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:05.971045 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxjpw\" (UniqueName: \"kubernetes.io/projected/7ede84b0-7533-415a-b73e-31454c67c06a-kube-api-access-pxjpw\") pod \"7ede84b0-7533-415a-b73e-31454c67c06a\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " Jul 24 04:00:05.971329 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:05.971073 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-istio-data\") pod \"7ede84b0-7533-415a-b73e-31454c67c06a\" (UID: \"7ede84b0-7533-415a-b73e-31454c67c06a\") " Jul 24 04:00:05.971329 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:05.971089 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7ede84b0-7533-415a-b73e-31454c67c06a" (UID: "7ede84b0-7533-415a-b73e-31454c67c06a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:00:05.971329 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:05.971179 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7ede84b0-7533-415a-b73e-31454c67c06a" (UID: "7ede84b0-7533-415a-b73e-31454c67c06a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:00:05.971329 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:05.971199 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7ede84b0-7533-415a-b73e-31454c67c06a" (UID: "7ede84b0-7533-415a-b73e-31454c67c06a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:00:05.971586 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:05.971414 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-credential-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:00:05.971586 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:05.971433 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-workload-certs\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:00:05.971586 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:05.971452 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-workload-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:00:05.971681 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:05.971580 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ede84b0-7533-415a-b73e-31454c67c06a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7ede84b0-7533-415a-b73e-31454c67c06a" (UID: "7ede84b0-7533-415a-b73e-31454c67c06a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 04:00:05.972083 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:05.972030 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-istio-data" (OuterVolumeSpecName: "istio-data") pod "7ede84b0-7533-415a-b73e-31454c67c06a" (UID: "7ede84b0-7533-415a-b73e-31454c67c06a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:00:05.973283 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:05.973255 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ede84b0-7533-415a-b73e-31454c67c06a-istio-token" (OuterVolumeSpecName: "istio-token") pod "7ede84b0-7533-415a-b73e-31454c67c06a" (UID: "7ede84b0-7533-415a-b73e-31454c67c06a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 04:00:05.973667 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:05.973647 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ede84b0-7533-415a-b73e-31454c67c06a-kube-api-access-pxjpw" (OuterVolumeSpecName: "kube-api-access-pxjpw") pod "7ede84b0-7533-415a-b73e-31454c67c06a" (UID: "7ede84b0-7533-415a-b73e-31454c67c06a"). InnerVolumeSpecName "kube-api-access-pxjpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 04:00:05.973758 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:05.973745 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7ede84b0-7533-415a-b73e-31454c67c06a" (UID: "7ede84b0-7533-415a-b73e-31454c67c06a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:00:05.973814 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:05.973789 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ede84b0-7533-415a-b73e-31454c67c06a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7ede84b0-7533-415a-b73e-31454c67c06a" (UID: "7ede84b0-7533-415a-b73e-31454c67c06a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 04:00:06.073007 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:06.072971 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ede84b0-7533-415a-b73e-31454c67c06a-istiod-ca-cert\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:00:06.073007 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:06.073000 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxjpw\" (UniqueName: \"kubernetes.io/projected/7ede84b0-7533-415a-b73e-31454c67c06a-kube-api-access-pxjpw\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:00:06.073007 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:06.073010 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-istio-data\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:00:06.073007 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:06.073018 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ede84b0-7533-415a-b73e-31454c67c06a-istio-envoy\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:00:06.073271 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:06.073026 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ede84b0-7533-415a-b73e-31454c67c06a-istio-podinfo\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:00:06.073271 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:06.073034 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ede84b0-7533-415a-b73e-31454c67c06a-istio-token\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:00:06.507050 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:06.507017 2579 generic.go:356] "Generic (PLEG): container finished" podID="7ede84b0-7533-415a-b73e-31454c67c06a" containerID="fc57c4adfbaf58cc1e3a1d9e4a5ecfa6a03275bc1ad1e482ba750ecbf8bb3303" exitCode=0 Jul 24 04:00:06.507488 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:06.507061 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" Jul 24 04:00:06.507488 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:06.507090 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" event={"ID":"7ede84b0-7533-415a-b73e-31454c67c06a","Type":"ContainerDied","Data":"fc57c4adfbaf58cc1e3a1d9e4a5ecfa6a03275bc1ad1e482ba750ecbf8bb3303"} Jul 24 04:00:06.507488 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:06.507135 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl" event={"ID":"7ede84b0-7533-415a-b73e-31454c67c06a","Type":"ContainerDied","Data":"2b5530bdc782f5fb2a44d019222cb0d05d250f48596a5b483269530bcb03f392"} Jul 24 04:00:06.507488 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:06.507158 2579 scope.go:117] "RemoveContainer" containerID="fc57c4adfbaf58cc1e3a1d9e4a5ecfa6a03275bc1ad1e482ba750ecbf8bb3303" Jul 24 04:00:06.520028 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:06.520007 2579 scope.go:117] "RemoveContainer" containerID="bdf902c539d94c5cf37b4e6dadd862f203dcf158048f87e0fe0b98044ad620de" Jul 24 04:00:06.528856 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:06.528783 2579 scope.go:117] "RemoveContainer" containerID="fc57c4adfbaf58cc1e3a1d9e4a5ecfa6a03275bc1ad1e482ba750ecbf8bb3303" Jul 24 04:00:06.529262 ip-10-0-134-147 kubenswrapper[2579]: E0724 04:00:06.529119 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc57c4adfbaf58cc1e3a1d9e4a5ecfa6a03275bc1ad1e482ba750ecbf8bb3303\": container with ID starting with fc57c4adfbaf58cc1e3a1d9e4a5ecfa6a03275bc1ad1e482ba750ecbf8bb3303 not found: ID does not exist" containerID="fc57c4adfbaf58cc1e3a1d9e4a5ecfa6a03275bc1ad1e482ba750ecbf8bb3303" Jul 24 04:00:06.529262 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:06.529157 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc57c4adfbaf58cc1e3a1d9e4a5ecfa6a03275bc1ad1e482ba750ecbf8bb3303"} err="failed to get container status \"fc57c4adfbaf58cc1e3a1d9e4a5ecfa6a03275bc1ad1e482ba750ecbf8bb3303\": rpc error: code = NotFound desc = could not find container \"fc57c4adfbaf58cc1e3a1d9e4a5ecfa6a03275bc1ad1e482ba750ecbf8bb3303\": container with ID starting with fc57c4adfbaf58cc1e3a1d9e4a5ecfa6a03275bc1ad1e482ba750ecbf8bb3303 not found: ID does not exist" Jul 24 04:00:06.529262 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:06.529185 2579 scope.go:117] "RemoveContainer" containerID="bdf902c539d94c5cf37b4e6dadd862f203dcf158048f87e0fe0b98044ad620de" Jul 24 04:00:06.531048 ip-10-0-134-147 kubenswrapper[2579]: E0724 04:00:06.531025 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdf902c539d94c5cf37b4e6dadd862f203dcf158048f87e0fe0b98044ad620de\": container with ID starting with bdf902c539d94c5cf37b4e6dadd862f203dcf158048f87e0fe0b98044ad620de not found: ID does not exist" containerID="bdf902c539d94c5cf37b4e6dadd862f203dcf158048f87e0fe0b98044ad620de" Jul 24 04:00:06.531154 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:06.531056 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdf902c539d94c5cf37b4e6dadd862f203dcf158048f87e0fe0b98044ad620de"} err="failed to get container status \"bdf902c539d94c5cf37b4e6dadd862f203dcf158048f87e0fe0b98044ad620de\": rpc error: code = NotFound desc = could not find container \"bdf902c539d94c5cf37b4e6dadd862f203dcf158048f87e0fe0b98044ad620de\": container with ID starting with bdf902c539d94c5cf37b4e6dadd862f203dcf158048f87e0fe0b98044ad620de not found: ID does not exist" Jul 24 04:00:06.532143 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:06.532121 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl"] Jul 24 04:00:06.536836 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:06.536810 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f822cd-openshift-default-55ff67559d-fx8zl"] Jul 24 04:00:07.034342 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:07.034308 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ede84b0-7533-415a-b73e-31454c67c06a" path="/var/lib/kubelet/pods/7ede84b0-7533-415a-b73e-31454c67c06a/volumes" Jul 24 04:00:12.091972 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:12.091927 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:26.840687 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:26.840627 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:27.350713 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:27.350675 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:27.411232 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:27.411198 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:30.131495 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:30.131426 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:31.352035 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:31.352000 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:31.452627 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:31.452593 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:37.321631 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:37.321592 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:37.443580 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:37.443543 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:37.637789 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:37.637758 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg"] Jul 24 04:00:37.638037 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:37.638012 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" podUID="41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd" containerName="istio-proxy" containerID="cri-o://f9eed29caefddaa2747a84aade44652a7b73a02e9b3a71e5807edd79a781b7f6" gracePeriod=30 Jul 24 04:00:37.655691 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:37.655660 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:37.862057 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:37.862023 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:38.300952 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.300914 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:38.600704 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.600619 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt"] Jul 24 04:00:38.601306 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.601289 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ede84b0-7533-415a-b73e-31454c67c06a" containerName="istio-proxy" Jul 24 04:00:38.601373 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.601308 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ede84b0-7533-415a-b73e-31454c67c06a" containerName="istio-proxy" Jul 24 04:00:38.601373 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.601358 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d10edbd0-e449-459f-b955-6950c7142712" containerName="istio-proxy" Jul 24 04:00:38.601373 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.601367 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d10edbd0-e449-459f-b955-6950c7142712" containerName="istio-proxy" Jul 24 04:00:38.601576 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.601497 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ede84b0-7533-415a-b73e-31454c67c06a" containerName="istio-proxy" Jul 24 04:00:38.601576 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.601512 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ede84b0-7533-415a-b73e-31454c67c06a" containerName="istio-proxy" Jul 24 04:00:38.601576 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.601526 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d10edbd0-e449-459f-b955-6950c7142712" containerName="istio-proxy" Jul 24 04:00:38.604997 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.604976 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:38.609421 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.609402 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-5b56b9-openshift-default-dockercfg-fpk96\"" Jul 24 04:00:38.618380 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.618358 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt"] Jul 24 04:00:38.637031 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.637004 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:38.690679 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.690648 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ad2308-505c-4c87-9e7b-6a6508b52375-istiod-ca-cert\") pod \"e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:38.690878 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.690708 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-workload-socket\") pod \"e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:38.690878 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.690741 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-istio-data\") pod \"e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:38.690878 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.690782 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ad2308-505c-4c87-9e7b-6a6508b52375-istio-podinfo\") pod \"e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:38.690878 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.690825 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-istio-envoy\") pod \"e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:38.690878 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.690849 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzc9z\" (UniqueName: \"kubernetes.io/projected/39ad2308-505c-4c87-9e7b-6a6508b52375-kube-api-access-bzc9z\") pod \"e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:38.691104 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.690913 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39ad2308-505c-4c87-9e7b-6a6508b52375-istio-token\") pod \"e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:38.691104 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.690968 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-credential-socket\") pod \"e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:38.691104 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.691023 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-workload-certs\") pod \"e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:38.792407 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.792365 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ad2308-505c-4c87-9e7b-6a6508b52375-istiod-ca-cert\") pod \"e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:38.792622 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.792439 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-workload-socket\") pod \"e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:38.792622 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.792501 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-istio-data\") pod \"e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:38.792622 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.792547 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ad2308-505c-4c87-9e7b-6a6508b52375-istio-podinfo\") pod \"e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:38.792622 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.792579 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-istio-envoy\") pod \"e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:38.792622 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.792604 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzc9z\" (UniqueName: \"kubernetes.io/projected/39ad2308-505c-4c87-9e7b-6a6508b52375-kube-api-access-bzc9z\") pod \"e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:38.792904 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.792638 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39ad2308-505c-4c87-9e7b-6a6508b52375-istio-token\") pod \"e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:38.792904 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.792674 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-credential-socket\") pod \"e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:38.792904 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.792708 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-workload-certs\") pod \"e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:38.793222 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.793178 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-workload-socket\") pod \"e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:38.793222 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.793211 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-istio-data\") pod \"e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:38.793563 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.793224 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-credential-socket\") pod \"e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:38.793563 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.793212 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ad2308-505c-4c87-9e7b-6a6508b52375-istiod-ca-cert\") pod \"e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:38.793563 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.793389 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-workload-certs\") pod \"e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:38.795126 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.795090 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:38.795447 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.795425 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-istio-envoy\") pod \"e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:38.795658 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.795640 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ad2308-505c-4c87-9e7b-6a6508b52375-istio-podinfo\") pod \"e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:38.802220 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.802187 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39ad2308-505c-4c87-9e7b-6a6508b52375-istio-token\") pod \"e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:38.802321 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.802310 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzc9z\" (UniqueName: \"kubernetes.io/projected/39ad2308-505c-4c87-9e7b-6a6508b52375-kube-api-access-bzc9z\") pod \"e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:38.916321 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:38.916293 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:39.057796 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:39.057774 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt"] Jul 24 04:00:39.059151 ip-10-0-134-147 kubenswrapper[2579]: W0724 04:00:39.059123 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ad2308_505c_4c87_9e7b_6a6508b52375.slice/crio-9c8845c351d4129008c40154c330b4d428866764c23a283e2267d8a715ea994d WatchSource:0}: Error finding container 9c8845c351d4129008c40154c330b4d428866764c23a283e2267d8a715ea994d: Status 404 returned error can't find the container with id 9c8845c351d4129008c40154c330b4d428866764c23a283e2267d8a715ea994d Jul 24 04:00:39.061330 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:39.061289 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:00:39.061409 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:39.061376 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:00:39.061448 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:39.061415 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:00:39.675126 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:39.675087 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" event={"ID":"39ad2308-505c-4c87-9e7b-6a6508b52375","Type":"ContainerStarted","Data":"87e19458f5bbdae659ce77616dc0522e0eef706d9a5ef7b3ba2f7ed93a327c48"} Jul 24 04:00:39.675526 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:39.675156 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" event={"ID":"39ad2308-505c-4c87-9e7b-6a6508b52375","Type":"ContainerStarted","Data":"9c8845c351d4129008c40154c330b4d428866764c23a283e2267d8a715ea994d"} Jul 24 04:00:39.697821 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:39.697767 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" podStartSLOduration=1.6977513279999998 podStartE2EDuration="1.697751328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 04:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 04:00:39.696288088 +0000 UTC m=+2589.135707548" watchObservedRunningTime="2026-07-24 04:00:39.697751328 +0000 UTC m=+2589.137170788" Jul 24 04:00:39.916800 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:39.916765 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:40.921927 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:40.921898 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:41.686556 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:41.686522 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:41.687473 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:41.687436 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:41.690254 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:41.690229 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:41.798328 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:41.798277 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:41.892228 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:41.892181 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:42.074916 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:42.074828 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:42.427272 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:42.427239 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:42.755012 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:42.754982 2579 patch_prober.go:28] interesting pod/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 24 04:00:42.755148 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:42.755038 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" podUID="41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 24 04:00:42.778638 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:42.778598 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:42.893539 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:42.893513 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 04:00:42.938527 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:42.938502 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istio-envoy\") pod \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " Jul 24 04:00:42.938698 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:42.938548 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-workload-certs\") pod \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " Jul 24 04:00:42.938698 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:42.938583 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-credential-socket\") pod \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " Jul 24 04:00:42.938698 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:42.938610 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xx6j8\" (UniqueName: \"kubernetes.io/projected/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-kube-api-access-xx6j8\") pod \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " Jul 24 04:00:42.938698 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:42.938648 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istio-data\") pod \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " Jul 24 04:00:42.938698 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:42.938663 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istio-token\") pod \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " Jul 24 04:00:42.938698 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:42.938680 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istiod-ca-cert\") pod \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " Jul 24 04:00:42.939084 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:42.938730 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istio-podinfo\") pod \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " Jul 24 04:00:42.939084 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:42.938753 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-workload-socket\") pod \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\" (UID: \"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd\") " Jul 24 04:00:42.939084 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:42.938836 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd" (UID: "41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:00:42.939084 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:42.938859 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd" (UID: "41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:00:42.939289 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:42.939141 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd" (UID: "41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 04:00:42.939289 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:42.939157 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-credential-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:00:42.939289 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:42.939177 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-workload-certs\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:00:42.939445 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:42.939399 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd" (UID: "41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:00:42.940382 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:42.940347 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istio-data" (OuterVolumeSpecName: "istio-data") pod "41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd" (UID: "41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:00:42.941197 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:42.941161 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istio-token" (OuterVolumeSpecName: "istio-token") pod "41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd" (UID: "41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 04:00:42.941297 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:42.941256 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd" (UID: "41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:00:42.941357 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:42.941301 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd" (UID: "41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 04:00:42.941755 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:42.941731 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-kube-api-access-xx6j8" (OuterVolumeSpecName: "kube-api-access-xx6j8") pod "41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd" (UID: "41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd"). InnerVolumeSpecName "kube-api-access-xx6j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 04:00:43.039759 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:43.039684 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istio-envoy\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:00:43.039759 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:43.039711 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx6j8\" (UniqueName: \"kubernetes.io/projected/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-kube-api-access-xx6j8\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:00:43.039759 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:43.039725 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istio-data\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:00:43.039759 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:43.039737 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istio-token\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:00:43.039759 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:43.039750 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istiod-ca-cert\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:00:43.039759 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:43.039763 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-istio-podinfo\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:00:43.040056 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:43.039775 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd-workload-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:00:43.699502 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:43.699452 2579 generic.go:356] "Generic (PLEG): container finished" podID="41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd" containerID="f9eed29caefddaa2747a84aade44652a7b73a02e9b3a71e5807edd79a781b7f6" exitCode=0 Jul 24 04:00:43.700055 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:43.699512 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" Jul 24 04:00:43.700055 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:43.699505 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" event={"ID":"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd","Type":"ContainerDied","Data":"f9eed29caefddaa2747a84aade44652a7b73a02e9b3a71e5807edd79a781b7f6"} Jul 24 04:00:43.700055 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:43.699643 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg" event={"ID":"41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd","Type":"ContainerDied","Data":"b54e360b0e581e20bc99f895d3ffc207f46fe3808575cdbf1046b714c5195206"} Jul 24 04:00:43.700055 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:43.699676 2579 scope.go:117] "RemoveContainer" containerID="f9eed29caefddaa2747a84aade44652a7b73a02e9b3a71e5807edd79a781b7f6" Jul 24 04:00:43.708909 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:43.708894 2579 scope.go:117] "RemoveContainer" containerID="cf3bf87f9d033fb7de96ab8892416b07181e37f072f1c59af3b97554aaab42ed" Jul 24 04:00:43.716803 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:43.716786 2579 scope.go:117] "RemoveContainer" containerID="f9eed29caefddaa2747a84aade44652a7b73a02e9b3a71e5807edd79a781b7f6" Jul 24 04:00:43.717049 ip-10-0-134-147 kubenswrapper[2579]: E0724 04:00:43.717029 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9eed29caefddaa2747a84aade44652a7b73a02e9b3a71e5807edd79a781b7f6\": container with ID starting with f9eed29caefddaa2747a84aade44652a7b73a02e9b3a71e5807edd79a781b7f6 not found: ID does not exist" containerID="f9eed29caefddaa2747a84aade44652a7b73a02e9b3a71e5807edd79a781b7f6" Jul 24 04:00:43.717104 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:43.717061 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9eed29caefddaa2747a84aade44652a7b73a02e9b3a71e5807edd79a781b7f6"} err="failed to get container status \"f9eed29caefddaa2747a84aade44652a7b73a02e9b3a71e5807edd79a781b7f6\": rpc error: code = NotFound desc = could not find container \"f9eed29caefddaa2747a84aade44652a7b73a02e9b3a71e5807edd79a781b7f6\": container with ID starting with f9eed29caefddaa2747a84aade44652a7b73a02e9b3a71e5807edd79a781b7f6 not found: ID does not exist" Jul 24 04:00:43.717104 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:43.717078 2579 scope.go:117] "RemoveContainer" containerID="cf3bf87f9d033fb7de96ab8892416b07181e37f072f1c59af3b97554aaab42ed" Jul 24 04:00:43.717281 ip-10-0-134-147 kubenswrapper[2579]: E0724 04:00:43.717267 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf3bf87f9d033fb7de96ab8892416b07181e37f072f1c59af3b97554aaab42ed\": container with ID starting with cf3bf87f9d033fb7de96ab8892416b07181e37f072f1c59af3b97554aaab42ed not found: ID does not exist" containerID="cf3bf87f9d033fb7de96ab8892416b07181e37f072f1c59af3b97554aaab42ed" Jul 24 04:00:43.717320 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:43.717284 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf3bf87f9d033fb7de96ab8892416b07181e37f072f1c59af3b97554aaab42ed"} err="failed to get container status \"cf3bf87f9d033fb7de96ab8892416b07181e37f072f1c59af3b97554aaab42ed\": rpc error: code = NotFound desc = could not find container \"cf3bf87f9d033fb7de96ab8892416b07181e37f072f1c59af3b97554aaab42ed\": container with ID starting with cf3bf87f9d033fb7de96ab8892416b07181e37f072f1c59af3b97554aaab42ed not found: ID does not exist" Jul 24 04:00:43.728095 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:43.728070 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg"] Jul 24 04:00:43.734305 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:43.734281 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b23325-openshift-default-cfbcb595b-4cklg"] Jul 24 04:00:45.034324 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:45.034289 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd" path="/var/lib/kubelet/pods/41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd/volumes" Jul 24 04:00:46.164690 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:46.164657 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:46.496290 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:46.496185 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:46.820252 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:46.820171 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57"] Jul 24 04:00:46.820799 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:46.820783 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd" containerName="istio-proxy" Jul 24 04:00:46.820847 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:46.820802 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd" containerName="istio-proxy" Jul 24 04:00:46.820847 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:46.820821 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ede84b0-7533-415a-b73e-31454c67c06a" containerName="istio-proxy" Jul 24 04:00:46.820847 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:46.820829 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ede84b0-7533-415a-b73e-31454c67c06a" containerName="istio-proxy" Jul 24 04:00:46.820847 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:46.820840 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd" containerName="istio-proxy" Jul 24 04:00:46.820983 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:46.820848 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd" containerName="istio-proxy" Jul 24 04:00:46.820983 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:46.820952 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd" containerName="istio-proxy" Jul 24 04:00:46.820983 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:46.820966 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="41a9a9e2-5c88-46d6-afbe-c7deb58bb5bd" containerName="istio-proxy" Jul 24 04:00:46.826167 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:46.826143 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:46.828775 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:46.828753 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-f701e0-openshift-default-dockercfg-f8smc\"" Jul 24 04:00:46.837361 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:46.837341 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57"] Jul 24 04:00:46.923954 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:46.923915 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:46.978929 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:46.978880 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istio-podinfo\") pod \"e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:46.979133 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:46.978957 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istio-token\") pod \"e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:46.979133 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:46.979012 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-credential-socket\") pod \"e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:46.979133 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:46.979070 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-workload-certs\") pod \"e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:46.979133 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:46.979105 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istiod-ca-cert\") pod \"e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:46.979351 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:46.979247 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-workload-socket\") pod \"e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:46.979413 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:46.979349 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istio-data\") pod \"e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:46.979413 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:46.979384 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istio-envoy\") pod \"e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:46.979581 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:46.979419 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6s8m\" (UniqueName: \"kubernetes.io/projected/c4a8d26a-ac7f-4ea9-b909-4032697ad009-kube-api-access-c6s8m\") pod \"e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:47.080493 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:47.080382 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-workload-socket\") pod \"e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:47.080493 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:47.080438 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istio-data\") pod \"e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:47.080493 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:47.080485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istio-envoy\") pod \"e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:47.080783 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:47.080514 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6s8m\" (UniqueName: \"kubernetes.io/projected/c4a8d26a-ac7f-4ea9-b909-4032697ad009-kube-api-access-c6s8m\") pod \"e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:47.080783 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:47.080561 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istio-podinfo\") pod \"e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:47.080783 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:47.080588 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istio-token\") pod \"e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:47.080783 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:47.080629 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-credential-socket\") pod \"e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:47.080783 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:47.080655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-workload-certs\") pod \"e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:47.080783 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:47.080688 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istiod-ca-cert\") pod \"e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:47.081116 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:47.080850 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-workload-socket\") pod \"e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:47.081116 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:47.080909 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istio-data\") pod \"e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:47.081219 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:47.081143 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-credential-socket\") pod \"e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:47.081353 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:47.081335 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istiod-ca-cert\") pod \"e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:47.081430 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:47.081337 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-workload-certs\") pod \"e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:47.083520 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:47.083493 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istio-envoy\") pod \"e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:47.083520 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:47.083516 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istio-podinfo\") pod \"e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:47.089330 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:47.089308 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istio-token\") pod \"e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:47.089427 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:47.089360 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6s8m\" (UniqueName: \"kubernetes.io/projected/c4a8d26a-ac7f-4ea9-b909-4032697ad009-kube-api-access-c6s8m\") pod \"e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:47.138089 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:47.138061 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:47.255505 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:47.255432 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:47.288161 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:47.288138 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57"] Jul 24 04:00:47.289580 ip-10-0-134-147 kubenswrapper[2579]: W0724 04:00:47.289554 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a8d26a_ac7f_4ea9_b909_4032697ad009.slice/crio-5945bbef00b415a81007a8c84a4a44b98fbd07c2593ab332343774c6db006326 WatchSource:0}: Error finding container 5945bbef00b415a81007a8c84a4a44b98fbd07c2593ab332343774c6db006326: Status 404 returned error can't find the container with id 5945bbef00b415a81007a8c84a4a44b98fbd07c2593ab332343774c6db006326 Jul 24 04:00:47.292007 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:47.291959 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:00:47.292096 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:47.292062 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:00:47.292135 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:47.292109 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:00:47.722205 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:47.722168 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" event={"ID":"c4a8d26a-ac7f-4ea9-b909-4032697ad009","Type":"ContainerStarted","Data":"a1f5d773f7832753012c357ec03b52d521987487fbc2f8ac56dd2b6f69dc8e4f"} Jul 24 04:00:47.722394 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:47.722234 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" event={"ID":"c4a8d26a-ac7f-4ea9-b909-4032697ad009","Type":"ContainerStarted","Data":"5945bbef00b415a81007a8c84a4a44b98fbd07c2593ab332343774c6db006326"} Jul 24 04:00:47.788367 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:47.788286 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" podStartSLOduration=1.78826275 podStartE2EDuration="1.78826275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 04:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 04:00:47.784491438 +0000 UTC m=+2597.223910899" watchObservedRunningTime="2026-07-24 04:00:47.78826275 +0000 UTC m=+2597.227682211" Jul 24 04:00:48.138709 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:48.138682 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:48.143115 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:48.143090 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:48.727578 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:48.727546 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:48.728473 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:48.728440 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:00:49.769588 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:49.769546 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:49.862235 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:49.862200 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:50.033639 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:50.033549 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:50.590127 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:50.590097 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:50.736695 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:50.736651 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:54.237880 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:54.237854 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:00:54.761053 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:54.760977 2579 generic.go:356] "Generic (PLEG): container finished" podID="39ad2308-505c-4c87-9e7b-6a6508b52375" containerID="87e19458f5bbdae659ce77616dc0522e0eef706d9a5ef7b3ba2f7ed93a327c48" exitCode=0 Jul 24 04:00:54.761053 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:54.761030 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" event={"ID":"39ad2308-505c-4c87-9e7b-6a6508b52375","Type":"ContainerDied","Data":"87e19458f5bbdae659ce77616dc0522e0eef706d9a5ef7b3ba2f7ed93a327c48"} Jul 24 04:00:54.761446 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:54.761432 2579 scope.go:117] "RemoveContainer" containerID="87e19458f5bbdae659ce77616dc0522e0eef706d9a5ef7b3ba2f7ed93a327c48" Jul 24 04:00:54.762526 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:54.762498 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:00:54.762583 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:54.762571 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:00:54.762619 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:54.762601 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:00:55.771177 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:55.771143 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" event={"ID":"39ad2308-505c-4c87-9e7b-6a6508b52375","Type":"ContainerStarted","Data":"dfd1b00a60c3fcc212d847bf7e3ee75e8d1c12c12337d7926bad3a298bf69f68"} Jul 24 04:00:55.917006 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:55.916957 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:55.918261 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:55.918234 2579 patch_prober.go:28] interesting pod/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Jul 24 04:00:55.918395 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:55.918295 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" podUID="39ad2308-505c-4c87-9e7b-6a6508b52375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Jul 24 04:00:56.686503 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:56.686450 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:00:56.917280 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:56.917245 2579 patch_prober.go:28] interesting pod/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Jul 24 04:00:56.917900 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:56.917860 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" podUID="39ad2308-505c-4c87-9e7b-6a6508b52375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Jul 24 04:00:57.917657 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:57.917618 2579 patch_prober.go:28] interesting pod/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Jul 24 04:00:57.918062 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:57.917675 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" podUID="39ad2308-505c-4c87-9e7b-6a6508b52375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Jul 24 04:00:58.917135 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:58.917100 2579 patch_prober.go:28] interesting pod/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Jul 24 04:00:58.917343 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:58.917156 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" podUID="39ad2308-505c-4c87-9e7b-6a6508b52375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Jul 24 04:00:59.917049 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:59.917008 2579 patch_prober.go:28] interesting pod/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Jul 24 04:00:59.917710 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:00:59.917672 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" podUID="39ad2308-505c-4c87-9e7b-6a6508b52375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Jul 24 04:01:00.917806 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:01:00.917764 2579 patch_prober.go:28] interesting pod/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Jul 24 04:01:00.918289 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:01:00.917841 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" podUID="39ad2308-505c-4c87-9e7b-6a6508b52375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Jul 24 04:01:01.920977 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:01:01.920944 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:01:01.922002 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:01:01.921979 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:01:06.843039 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:01:06.843007 2579 generic.go:356] "Generic (PLEG): container finished" podID="c4a8d26a-ac7f-4ea9-b909-4032697ad009" containerID="a1f5d773f7832753012c357ec03b52d521987487fbc2f8ac56dd2b6f69dc8e4f" exitCode=0 Jul 24 04:01:06.843420 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:01:06.843078 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" event={"ID":"c4a8d26a-ac7f-4ea9-b909-4032697ad009","Type":"ContainerDied","Data":"a1f5d773f7832753012c357ec03b52d521987487fbc2f8ac56dd2b6f69dc8e4f"} Jul 24 04:01:06.843538 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:01:06.843524 2579 scope.go:117] "RemoveContainer" containerID="a1f5d773f7832753012c357ec03b52d521987487fbc2f8ac56dd2b6f69dc8e4f" Jul 24 04:01:06.844640 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:01:06.844611 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:01:06.844716 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:01:06.844699 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:01:06.844758 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:01:06.844746 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:01:07.850337 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:01:07.850298 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" event={"ID":"c4a8d26a-ac7f-4ea9-b909-4032697ad009","Type":"ContainerStarted","Data":"6ffe6e27f3323d4476ea6904237739ae99163c8731e79393f963129dbccdeb29"} Jul 24 04:01:08.139181 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:01:08.139096 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:01:08.139506 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:01:08.139476 2579 patch_prober.go:28] interesting pod/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 24 04:01:08.139679 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:01:08.139649 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" podUID="c4a8d26a-ac7f-4ea9-b909-4032697ad009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 24 04:01:09.138837 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:01:09.138800 2579 patch_prober.go:28] interesting pod/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 24 04:01:09.139333 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:01:09.138862 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" podUID="c4a8d26a-ac7f-4ea9-b909-4032697ad009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 24 04:01:10.138674 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:01:10.138639 2579 patch_prober.go:28] interesting pod/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 24 04:01:10.138858 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:01:10.138694 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" podUID="c4a8d26a-ac7f-4ea9-b909-4032697ad009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 24 04:01:11.139042 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:01:11.139005 2579 patch_prober.go:28] interesting pod/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 24 04:01:11.139454 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:01:11.139067 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" podUID="c4a8d26a-ac7f-4ea9-b909-4032697ad009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 24 04:01:12.138948 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:01:12.138911 2579 patch_prober.go:28] interesting pod/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Jul 24 04:01:12.139149 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:01:12.138984 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" podUID="c4a8d26a-ac7f-4ea9-b909-4032697ad009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Jul 24 04:01:13.142008 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:01:13.141976 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:01:13.142376 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:01:13.142342 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:01:13.143053 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:01:13.143032 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:01:15.299621 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:01:15.299583 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:01:45.071957 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:01:45.071899 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:02:43.816142 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:02:43.816106 2579 scope.go:117] "RemoveContainer" containerID="48748d119320bca2322b1e6c69b16c2d65a9e7bb51759a8b554fda360923c9ad" Jul 24 04:03:10.789782 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:03:10.789743 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:03:12.990221 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:03:12.990185 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:03:48.292948 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:03:48.292917 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:03:49.456055 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:03:49.456022 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:04:04.162701 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:04:04.162662 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:04:25.764215 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:04:25.764177 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:05:13.283737 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:05:13.283705 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:05:14.190016 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:05:14.189983 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:05:20.737745 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:05:20.737711 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:05:42.459958 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:05:42.459919 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:06:24.515984 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:24.515951 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:06:24.567254 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:24.567221 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:06:25.232509 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:25.232454 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:06:26.238432 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:26.238401 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:06:27.185599 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:27.185569 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7627-140150871926448"] Jul 24 04:06:27.188044 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:27.188026 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7627-140150871926448" Jul 24 04:06:27.190378 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:27.190363 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-sqpgk\"" Jul 24 04:06:27.195200 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:27.194800 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7627-140150871926448"] Jul 24 04:06:27.334085 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:27.334053 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsg88\" (UniqueName: \"kubernetes.io/projected/e2a20350-b105-48c1-a710-b0c3d5dd4110-kube-api-access-tsg88\") pod \"test-curl-7627-140150871926448\" (UID: \"e2a20350-b105-48c1-a710-b0c3d5dd4110\") " pod="opendatahub/test-curl-7627-140150871926448" Jul 24 04:06:27.435587 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:27.435555 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsg88\" (UniqueName: \"kubernetes.io/projected/e2a20350-b105-48c1-a710-b0c3d5dd4110-kube-api-access-tsg88\") pod \"test-curl-7627-140150871926448\" (UID: \"e2a20350-b105-48c1-a710-b0c3d5dd4110\") " pod="opendatahub/test-curl-7627-140150871926448" Jul 24 04:06:27.444067 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:27.444001 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsg88\" (UniqueName: \"kubernetes.io/projected/e2a20350-b105-48c1-a710-b0c3d5dd4110-kube-api-access-tsg88\") pod \"test-curl-7627-140150871926448\" (UID: \"e2a20350-b105-48c1-a710-b0c3d5dd4110\") " pod="opendatahub/test-curl-7627-140150871926448" Jul 24 04:06:27.497354 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:27.497319 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7627-140150871926448" Jul 24 04:06:27.631845 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:27.631817 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7627-140150871926448"] Jul 24 04:06:27.633842 ip-10-0-134-147 kubenswrapper[2579]: W0724 04:06:27.633810 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2a20350_b105_48c1_a710_b0c3d5dd4110.slice/crio-7d2e1adc9b3cb9b71fda9c47ed637e8dba0e21049148a9c09419ae0966f0911d WatchSource:0}: Error finding container 7d2e1adc9b3cb9b71fda9c47ed637e8dba0e21049148a9c09419ae0966f0911d: Status 404 returned error can't find the container with id 7d2e1adc9b3cb9b71fda9c47ed637e8dba0e21049148a9c09419ae0966f0911d Jul 24 04:06:27.635250 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:27.635228 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 04:06:28.499353 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:28.499321 2579 generic.go:356] "Generic (PLEG): container finished" podID="e2a20350-b105-48c1-a710-b0c3d5dd4110" containerID="9fce2c9a7fd0655d17a0fcd82d062a8ca7837ced045d2c611e0f3cb4c6aa21b0" exitCode=0 Jul 24 04:06:28.499720 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:28.499375 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7627-140150871926448" event={"ID":"e2a20350-b105-48c1-a710-b0c3d5dd4110","Type":"ContainerDied","Data":"9fce2c9a7fd0655d17a0fcd82d062a8ca7837ced045d2c611e0f3cb4c6aa21b0"} Jul 24 04:06:28.499720 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:28.499416 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7627-140150871926448" event={"ID":"e2a20350-b105-48c1-a710-b0c3d5dd4110","Type":"ContainerStarted","Data":"7d2e1adc9b3cb9b71fda9c47ed637e8dba0e21049148a9c09419ae0966f0911d"} Jul 24 04:06:29.639272 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:29.639246 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7627-140150871926448" Jul 24 04:06:29.659109 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:29.659088 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7627-140150871926448_e2a20350-b105-48c1-a710-b0c3d5dd4110/test-curl-7627-140150871926448/0.log" Jul 24 04:06:29.682567 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:29.682546 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7627-140150871926448"] Jul 24 04:06:29.686767 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:29.686750 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7627-140150871926448"] Jul 24 04:06:29.756962 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:29.756898 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsg88\" (UniqueName: \"kubernetes.io/projected/e2a20350-b105-48c1-a710-b0c3d5dd4110-kube-api-access-tsg88\") pod \"e2a20350-b105-48c1-a710-b0c3d5dd4110\" (UID: \"e2a20350-b105-48c1-a710-b0c3d5dd4110\") " Jul 24 04:06:29.759134 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:29.759101 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a20350-b105-48c1-a710-b0c3d5dd4110-kube-api-access-tsg88" (OuterVolumeSpecName: "kube-api-access-tsg88") pod "e2a20350-b105-48c1-a710-b0c3d5dd4110" (UID: "e2a20350-b105-48c1-a710-b0c3d5dd4110"). InnerVolumeSpecName "kube-api-access-tsg88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 04:06:29.808102 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:29.808077 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7627-140150871927472"] Jul 24 04:06:29.808521 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:29.808507 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2a20350-b105-48c1-a710-b0c3d5dd4110" containerName="test-curl-7627-140150871926448" Jul 24 04:06:29.808569 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:29.808522 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2a20350-b105-48c1-a710-b0c3d5dd4110" containerName="test-curl-7627-140150871926448" Jul 24 04:06:29.808605 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:29.808584 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2a20350-b105-48c1-a710-b0c3d5dd4110" containerName="test-curl-7627-140150871926448" Jul 24 04:06:29.811522 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:29.811507 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7627-140150871927472" Jul 24 04:06:29.817515 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:29.817492 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7627-140150871927472"] Jul 24 04:06:29.857970 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:29.857940 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsg88\" (UniqueName: \"kubernetes.io/projected/e2a20350-b105-48c1-a710-b0c3d5dd4110-kube-api-access-tsg88\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:06:29.959045 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:29.959006 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqx2k\" (UniqueName: \"kubernetes.io/projected/b2204f6a-3331-46b8-aa24-340cf0719687-kube-api-access-dqx2k\") pod \"test-curl-7627-140150871927472\" (UID: \"b2204f6a-3331-46b8-aa24-340cf0719687\") " pod="opendatahub/test-curl-7627-140150871927472" Jul 24 04:06:30.060924 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:30.060829 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqx2k\" (UniqueName: \"kubernetes.io/projected/b2204f6a-3331-46b8-aa24-340cf0719687-kube-api-access-dqx2k\") pod \"test-curl-7627-140150871927472\" (UID: \"b2204f6a-3331-46b8-aa24-340cf0719687\") " pod="opendatahub/test-curl-7627-140150871927472" Jul 24 04:06:30.069486 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:30.069425 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqx2k\" (UniqueName: \"kubernetes.io/projected/b2204f6a-3331-46b8-aa24-340cf0719687-kube-api-access-dqx2k\") pod \"test-curl-7627-140150871927472\" (UID: \"b2204f6a-3331-46b8-aa24-340cf0719687\") " pod="opendatahub/test-curl-7627-140150871927472" Jul 24 04:06:30.123175 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:30.123144 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7627-140150871927472" Jul 24 04:06:30.255651 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:30.255625 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7627-140150871927472"] Jul 24 04:06:30.257774 ip-10-0-134-147 kubenswrapper[2579]: W0724 04:06:30.257745 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2204f6a_3331_46b8_aa24_340cf0719687.slice/crio-29ca3882dab8cf9dc5a0f793506b402e6f3e8e62ef4b5ffb72d3a5557e9d1c11 WatchSource:0}: Error finding container 29ca3882dab8cf9dc5a0f793506b402e6f3e8e62ef4b5ffb72d3a5557e9d1c11: Status 404 returned error can't find the container with id 29ca3882dab8cf9dc5a0f793506b402e6f3e8e62ef4b5ffb72d3a5557e9d1c11 Jul 24 04:06:30.510295 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:30.510261 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7627-140150871927472" event={"ID":"b2204f6a-3331-46b8-aa24-340cf0719687","Type":"ContainerStarted","Data":"29ca3882dab8cf9dc5a0f793506b402e6f3e8e62ef4b5ffb72d3a5557e9d1c11"} Jul 24 04:06:30.511555 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:30.511534 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d2e1adc9b3cb9b71fda9c47ed637e8dba0e21049148a9c09419ae0966f0911d" Jul 24 04:06:30.511647 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:30.511566 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7627-140150871926448" Jul 24 04:06:30.521684 ip-10-0-134-147 kubenswrapper[2579]: E0724 04:06:30.521656 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7627-140150871926448\" is forbidden: User \"system:node:ip-10-0-134-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-147.ec2.internal' and this object" podUID="e2a20350-b105-48c1-a710-b0c3d5dd4110" pod="opendatahub/test-curl-7627-140150871926448" Jul 24 04:06:31.032420 ip-10-0-134-147 kubenswrapper[2579]: E0724 04:06:31.032383 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7627-140150871926448\" is forbidden: User \"system:node:ip-10-0-134-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-147.ec2.internal' and this object" podUID="e2a20350-b105-48c1-a710-b0c3d5dd4110" pod="opendatahub/test-curl-7627-140150871926448" Jul 24 04:06:31.034244 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:31.034221 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2a20350-b105-48c1-a710-b0c3d5dd4110" path="/var/lib/kubelet/pods/e2a20350-b105-48c1-a710-b0c3d5dd4110/volumes" Jul 24 04:06:31.525479 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:31.525428 2579 generic.go:356] "Generic (PLEG): container finished" podID="b2204f6a-3331-46b8-aa24-340cf0719687" containerID="5537310034de52003a2b782ad0ed06a9906eea5ca6667e5e98b1a2c12b49e653" exitCode=0 Jul 24 04:06:31.525685 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:31.525513 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7627-140150871927472" event={"ID":"b2204f6a-3331-46b8-aa24-340cf0719687","Type":"ContainerDied","Data":"5537310034de52003a2b782ad0ed06a9906eea5ca6667e5e98b1a2c12b49e653"} Jul 24 04:06:32.666672 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:32.666648 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7627-140150871927472" Jul 24 04:06:32.687532 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:32.687501 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7627-140150871927472_b2204f6a-3331-46b8-aa24-340cf0719687/test-curl-7627-140150871927472/0.log" Jul 24 04:06:32.713707 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:32.713676 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7627-140150871927472"] Jul 24 04:06:32.717012 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:32.716988 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7627-140150871927472"] Jul 24 04:06:32.788268 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:32.788174 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqx2k\" (UniqueName: \"kubernetes.io/projected/b2204f6a-3331-46b8-aa24-340cf0719687-kube-api-access-dqx2k\") pod \"b2204f6a-3331-46b8-aa24-340cf0719687\" (UID: \"b2204f6a-3331-46b8-aa24-340cf0719687\") " Jul 24 04:06:32.790350 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:32.790321 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2204f6a-3331-46b8-aa24-340cf0719687-kube-api-access-dqx2k" (OuterVolumeSpecName: "kube-api-access-dqx2k") pod "b2204f6a-3331-46b8-aa24-340cf0719687" (UID: "b2204f6a-3331-46b8-aa24-340cf0719687"). InnerVolumeSpecName "kube-api-access-dqx2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 04:06:32.877786 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:32.877753 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7627-140150868692832"] Jul 24 04:06:32.878215 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:32.878202 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2204f6a-3331-46b8-aa24-340cf0719687" containerName="test-curl-7627-140150871927472" Jul 24 04:06:32.878262 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:32.878217 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2204f6a-3331-46b8-aa24-340cf0719687" containerName="test-curl-7627-140150871927472" Jul 24 04:06:32.878297 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:32.878289 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2204f6a-3331-46b8-aa24-340cf0719687" containerName="test-curl-7627-140150871927472" Jul 24 04:06:32.881382 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:32.881367 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7627-140150868692832" Jul 24 04:06:32.885604 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:32.885582 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7627-140150868692832"] Jul 24 04:06:32.889292 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:32.889267 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqx2k\" (UniqueName: \"kubernetes.io/projected/b2204f6a-3331-46b8-aa24-340cf0719687-kube-api-access-dqx2k\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:06:32.989996 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:32.989962 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkf84\" (UniqueName: \"kubernetes.io/projected/99e33184-96b7-46b0-bac9-916f4e75af70-kube-api-access-dkf84\") pod \"test-curl-7627-140150868692832\" (UID: \"99e33184-96b7-46b0-bac9-916f4e75af70\") " pod="opendatahub/test-curl-7627-140150868692832" Jul 24 04:06:33.034594 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:33.034561 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2204f6a-3331-46b8-aa24-340cf0719687" path="/var/lib/kubelet/pods/b2204f6a-3331-46b8-aa24-340cf0719687/volumes" Jul 24 04:06:33.091440 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:33.091411 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkf84\" (UniqueName: \"kubernetes.io/projected/99e33184-96b7-46b0-bac9-916f4e75af70-kube-api-access-dkf84\") pod \"test-curl-7627-140150868692832\" (UID: \"99e33184-96b7-46b0-bac9-916f4e75af70\") " pod="opendatahub/test-curl-7627-140150868692832" Jul 24 04:06:33.099909 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:33.099884 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkf84\" (UniqueName: \"kubernetes.io/projected/99e33184-96b7-46b0-bac9-916f4e75af70-kube-api-access-dkf84\") pod \"test-curl-7627-140150868692832\" (UID: \"99e33184-96b7-46b0-bac9-916f4e75af70\") " pod="opendatahub/test-curl-7627-140150868692832" Jul 24 04:06:33.194322 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:33.194279 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7627-140150868692832" Jul 24 04:06:33.328779 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:33.328755 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7627-140150868692832"] Jul 24 04:06:33.330793 ip-10-0-134-147 kubenswrapper[2579]: W0724 04:06:33.330771 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99e33184_96b7_46b0_bac9_916f4e75af70.slice/crio-394cd58d237daf5eab8afe8bbc80a1e1ae086a931acb73c9cad42b0d01690d36 WatchSource:0}: Error finding container 394cd58d237daf5eab8afe8bbc80a1e1ae086a931acb73c9cad42b0d01690d36: Status 404 returned error can't find the container with id 394cd58d237daf5eab8afe8bbc80a1e1ae086a931acb73c9cad42b0d01690d36 Jul 24 04:06:33.536216 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:33.536191 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7627-140150871927472" Jul 24 04:06:33.536409 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:33.536246 2579 scope.go:117] "RemoveContainer" containerID="5537310034de52003a2b782ad0ed06a9906eea5ca6667e5e98b1a2c12b49e653" Jul 24 04:06:33.537645 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:33.537619 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7627-140150868692832" event={"ID":"99e33184-96b7-46b0-bac9-916f4e75af70","Type":"ContainerStarted","Data":"394cd58d237daf5eab8afe8bbc80a1e1ae086a931acb73c9cad42b0d01690d36"} Jul 24 04:06:34.545148 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:34.545110 2579 generic.go:356] "Generic (PLEG): container finished" podID="99e33184-96b7-46b0-bac9-916f4e75af70" containerID="2775a8bac9bd5ec2b666160840a0976e405445ab51312b07ea35b9484c2f6325" exitCode=0 Jul 24 04:06:34.545613 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:34.545159 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7627-140150868692832" event={"ID":"99e33184-96b7-46b0-bac9-916f4e75af70","Type":"ContainerDied","Data":"2775a8bac9bd5ec2b666160840a0976e405445ab51312b07ea35b9484c2f6325"} Jul 24 04:06:35.692497 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:35.692445 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7627-140150868692832" Jul 24 04:06:35.713399 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:35.713376 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7627-140150868692832_99e33184-96b7-46b0-bac9-916f4e75af70/test-curl-7627-140150868692832/0.log" Jul 24 04:06:35.743120 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:35.743082 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7627-140150868692832"] Jul 24 04:06:35.745067 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:35.745049 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7627-140150868692832"] Jul 24 04:06:35.817595 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:35.817527 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkf84\" (UniqueName: \"kubernetes.io/projected/99e33184-96b7-46b0-bac9-916f4e75af70-kube-api-access-dkf84\") pod \"99e33184-96b7-46b0-bac9-916f4e75af70\" (UID: \"99e33184-96b7-46b0-bac9-916f4e75af70\") " Jul 24 04:06:35.819737 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:35.819707 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e33184-96b7-46b0-bac9-916f4e75af70-kube-api-access-dkf84" (OuterVolumeSpecName: "kube-api-access-dkf84") pod "99e33184-96b7-46b0-bac9-916f4e75af70" (UID: "99e33184-96b7-46b0-bac9-916f4e75af70"). InnerVolumeSpecName "kube-api-access-dkf84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 04:06:35.918397 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:35.918365 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkf84\" (UniqueName: \"kubernetes.io/projected/99e33184-96b7-46b0-bac9-916f4e75af70-kube-api-access-dkf84\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:06:36.556490 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:36.556444 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7627-140150868692832" Jul 24 04:06:36.556490 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:36.556485 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="394cd58d237daf5eab8afe8bbc80a1e1ae086a931acb73c9cad42b0d01690d36" Jul 24 04:06:36.566887 ip-10-0-134-147 kubenswrapper[2579]: E0724 04:06:36.566858 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7627-140150868692832\" is forbidden: User \"system:node:ip-10-0-134-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-147.ec2.internal' and this object" podUID="99e33184-96b7-46b0-bac9-916f4e75af70" pod="opendatahub/test-curl-7627-140150868692832" Jul 24 04:06:37.034584 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:37.034546 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99e33184-96b7-46b0-bac9-916f4e75af70" path="/var/lib/kubelet/pods/99e33184-96b7-46b0-bac9-916f4e75af70/volumes" Jul 24 04:06:38.972904 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:38.972874 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7627-140150871897552"] Jul 24 04:06:38.973360 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:38.973346 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99e33184-96b7-46b0-bac9-916f4e75af70" containerName="test-curl-7627-140150868692832" Jul 24 04:06:38.973421 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:38.973362 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="99e33184-96b7-46b0-bac9-916f4e75af70" containerName="test-curl-7627-140150868692832" Jul 24 04:06:38.973486 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:38.973450 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="99e33184-96b7-46b0-bac9-916f4e75af70" containerName="test-curl-7627-140150868692832" Jul 24 04:06:38.976579 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:38.976563 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7627-140150871897552" Jul 24 04:06:38.982476 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:38.982432 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-sqpgk\"" Jul 24 04:06:38.985511 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:38.985487 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7627-140150871897552"] Jul 24 04:06:39.148160 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:39.148130 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56s4b\" (UniqueName: \"kubernetes.io/projected/3d1c720a-d5a9-4c41-a655-5f097e71a5f7-kube-api-access-56s4b\") pod \"test-curl-7627-140150871897552\" (UID: \"3d1c720a-d5a9-4c41-a655-5f097e71a5f7\") " pod="opendatahub/test-curl-7627-140150871897552" Jul 24 04:06:39.249501 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:39.249392 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56s4b\" (UniqueName: \"kubernetes.io/projected/3d1c720a-d5a9-4c41-a655-5f097e71a5f7-kube-api-access-56s4b\") pod \"test-curl-7627-140150871897552\" (UID: \"3d1c720a-d5a9-4c41-a655-5f097e71a5f7\") " pod="opendatahub/test-curl-7627-140150871897552" Jul 24 04:06:39.258650 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:39.258622 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56s4b\" (UniqueName: \"kubernetes.io/projected/3d1c720a-d5a9-4c41-a655-5f097e71a5f7-kube-api-access-56s4b\") pod \"test-curl-7627-140150871897552\" (UID: \"3d1c720a-d5a9-4c41-a655-5f097e71a5f7\") " pod="opendatahub/test-curl-7627-140150871897552" Jul 24 04:06:39.287177 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:39.287137 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7627-140150871897552" Jul 24 04:06:39.425791 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:39.425765 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7627-140150871897552"] Jul 24 04:06:39.428126 ip-10-0-134-147 kubenswrapper[2579]: W0724 04:06:39.428098 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d1c720a_d5a9_4c41_a655_5f097e71a5f7.slice/crio-f2917932c0c62edb6af66452b984e8672dca01ed407d01699821434747381148 WatchSource:0}: Error finding container f2917932c0c62edb6af66452b984e8672dca01ed407d01699821434747381148: Status 404 returned error can't find the container with id f2917932c0c62edb6af66452b984e8672dca01ed407d01699821434747381148 Jul 24 04:06:39.574658 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:39.574568 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7627-140150871897552" event={"ID":"3d1c720a-d5a9-4c41-a655-5f097e71a5f7","Type":"ContainerStarted","Data":"f2917932c0c62edb6af66452b984e8672dca01ed407d01699821434747381148"} Jul 24 04:06:40.581079 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:40.580986 2579 generic.go:356] "Generic (PLEG): container finished" podID="3d1c720a-d5a9-4c41-a655-5f097e71a5f7" containerID="3a96d1b4c4b7ecfb5352d16b4f6a9a5183ddeb5c030940c2ef0e42405a7fe9fb" exitCode=0 Jul 24 04:06:40.581079 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:40.581069 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7627-140150871897552" event={"ID":"3d1c720a-d5a9-4c41-a655-5f097e71a5f7","Type":"ContainerDied","Data":"3a96d1b4c4b7ecfb5352d16b4f6a9a5183ddeb5c030940c2ef0e42405a7fe9fb"} Jul 24 04:06:41.730250 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:41.730225 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7627-140150871897552" Jul 24 04:06:41.756379 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:41.756274 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7627-140150871897552_3d1c720a-d5a9-4c41-a655-5f097e71a5f7/test-curl-7627-140150871897552/0.log" Jul 24 04:06:41.781753 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:41.781729 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7627-140150871897552"] Jul 24 04:06:41.788487 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:41.788452 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7627-140150871897552"] Jul 24 04:06:41.876000 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:41.875975 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56s4b\" (UniqueName: \"kubernetes.io/projected/3d1c720a-d5a9-4c41-a655-5f097e71a5f7-kube-api-access-56s4b\") pod \"3d1c720a-d5a9-4c41-a655-5f097e71a5f7\" (UID: \"3d1c720a-d5a9-4c41-a655-5f097e71a5f7\") " Jul 24 04:06:41.878191 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:41.878160 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1c720a-d5a9-4c41-a655-5f097e71a5f7-kube-api-access-56s4b" (OuterVolumeSpecName: "kube-api-access-56s4b") pod "3d1c720a-d5a9-4c41-a655-5f097e71a5f7" (UID: "3d1c720a-d5a9-4c41-a655-5f097e71a5f7"). InnerVolumeSpecName "kube-api-access-56s4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 04:06:41.976891 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:41.976857 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56s4b\" (UniqueName: \"kubernetes.io/projected/3d1c720a-d5a9-4c41-a655-5f097e71a5f7-kube-api-access-56s4b\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:06:42.592896 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:42.592868 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7627-140150871897552" Jul 24 04:06:42.593051 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:42.592906 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2917932c0c62edb6af66452b984e8672dca01ed407d01699821434747381148" Jul 24 04:06:42.602352 ip-10-0-134-147 kubenswrapper[2579]: E0724 04:06:42.602321 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7627-140150871897552\" is forbidden: User \"system:node:ip-10-0-134-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-147.ec2.internal' and this object" podUID="3d1c720a-d5a9-4c41-a655-5f097e71a5f7" pod="opendatahub/test-curl-7627-140150871897552" Jul 24 04:06:43.034546 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:43.034518 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d1c720a-d5a9-4c41-a655-5f097e71a5f7" path="/var/lib/kubelet/pods/3d1c720a-d5a9-4c41-a655-5f097e71a5f7/volumes" Jul 24 04:06:45.464604 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:45.464569 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:06:45.805447 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:45.805375 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp"] Jul 24 04:06:45.805876 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:45.805860 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d1c720a-d5a9-4c41-a655-5f097e71a5f7" containerName="test-curl-7627-140150871897552" Jul 24 04:06:45.805923 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:45.805878 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d1c720a-d5a9-4c41-a655-5f097e71a5f7" containerName="test-curl-7627-140150871897552" Jul 24 04:06:45.805962 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:45.805953 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d1c720a-d5a9-4c41-a655-5f097e71a5f7" containerName="test-curl-7627-140150871897552" Jul 24 04:06:45.810233 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:45.810212 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:45.813976 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:45.813961 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-733dde-openshift-default-dockercfg-x48sp\"" Jul 24 04:06:45.823995 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:45.823973 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp"] Jul 24 04:06:45.852828 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:45.852800 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:06:45.915028 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:45.914946 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-workload-certs\") pod \"e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:45.915028 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:45.914996 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-workload-socket\") pod \"e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:45.915336 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:45.915066 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntq6k\" (UniqueName: \"kubernetes.io/projected/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-kube-api-access-ntq6k\") pod \"e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:45.915336 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:45.915140 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istio-envoy\") pod \"e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:45.915336 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:45.915174 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-credential-socket\") pod \"e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:45.915336 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:45.915216 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istio-data\") pod \"e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:45.915336 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:45.915243 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istio-token\") pod \"e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:45.915336 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:45.915276 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istio-podinfo\") pod \"e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:45.915336 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:45.915301 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istiod-ca-cert\") pod \"e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:46.016980 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:46.016943 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istio-envoy\") pod \"e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:46.017167 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:46.016996 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-credential-socket\") pod \"e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:46.017167 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:46.017034 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istio-data\") pod \"e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:46.017167 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:46.017060 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istio-token\") pod \"e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:46.017167 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:46.017090 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istio-podinfo\") pod \"e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:46.017167 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:46.017114 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istiod-ca-cert\") pod \"e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:46.017424 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:46.017189 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-workload-certs\") pod \"e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:46.017424 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:46.017217 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-workload-socket\") pod \"e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:46.017424 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:46.017277 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntq6k\" (UniqueName: \"kubernetes.io/projected/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-kube-api-access-ntq6k\") pod \"e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:46.017625 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:46.017489 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istio-data\") pod \"e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:46.017625 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:46.017571 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-workload-certs\") pod \"e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:46.017802 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:46.017726 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-workload-socket\") pod \"e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:46.017953 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:46.017934 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-credential-socket\") pod \"e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:46.018297 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:46.018275 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istiod-ca-cert\") pod \"e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:46.019517 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:46.019495 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istio-envoy\") pod \"e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:46.019724 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:46.019706 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istio-podinfo\") pod \"e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:46.026570 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:46.026540 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntq6k\" (UniqueName: \"kubernetes.io/projected/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-kube-api-access-ntq6k\") pod \"e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:46.027140 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:46.027119 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istio-token\") pod \"e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:46.121515 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:46.121483 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:46.135737 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:46.135706 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:06:46.262304 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:46.262277 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp"] Jul 24 04:06:46.264188 ip-10-0-134-147 kubenswrapper[2579]: W0724 04:06:46.264165 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a38ba5_6d7b_4dd9_87ec_1992ba1f2c57.slice/crio-7cb6e4696b711de256aec5767940851d38fc7951f1bba90b2d953a4659bd4190 WatchSource:0}: Error finding container 7cb6e4696b711de256aec5767940851d38fc7951f1bba90b2d953a4659bd4190: Status 404 returned error can't find the container with id 7cb6e4696b711de256aec5767940851d38fc7951f1bba90b2d953a4659bd4190 Jul 24 04:06:46.266811 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:46.266782 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:06:46.266899 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:46.266862 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:06:46.266966 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:46.266918 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:06:46.615845 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:46.615800 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" event={"ID":"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57","Type":"ContainerStarted","Data":"5162645ce5442392652579e965b52ce18098996a452bf5c7e362dfe25c53992a"} Jul 24 04:06:46.616240 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:46.615869 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" event={"ID":"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57","Type":"ContainerStarted","Data":"7cb6e4696b711de256aec5767940851d38fc7951f1bba90b2d953a4659bd4190"} Jul 24 04:06:46.633842 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:46.633793 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" podStartSLOduration=1.633776664 podStartE2EDuration="1.633776664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 04:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 04:06:46.632792973 +0000 UTC m=+2956.072212448" watchObservedRunningTime="2026-07-24 04:06:46.633776664 +0000 UTC m=+2956.073196123" Jul 24 04:06:47.122638 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:47.122604 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:47.127342 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:47.127311 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:47.621917 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:47.621889 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:47.622857 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:47.622837 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:06:48.611426 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:48.611384 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:06:48.715261 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:48.715223 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:06:48.846881 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:48.846848 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:06:49.210270 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:49.210237 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:06:49.683865 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:49.683836 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:06:49.771577 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:49.771544 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:06:53.077279 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:06:53.077193 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:07:02.623333 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:02.623295 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Jul 24 04:07:02.623775 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:02.623366 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" podUID="07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Jul 24 04:07:02.713678 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:02.713646 2579 generic.go:356] "Generic (PLEG): container finished" podID="07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57" containerID="5162645ce5442392652579e965b52ce18098996a452bf5c7e362dfe25c53992a" exitCode=0 Jul 24 04:07:02.713850 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:02.713721 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" event={"ID":"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57","Type":"ContainerDied","Data":"5162645ce5442392652579e965b52ce18098996a452bf5c7e362dfe25c53992a"} Jul 24 04:07:02.714200 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:02.714185 2579 scope.go:117] "RemoveContainer" containerID="5162645ce5442392652579e965b52ce18098996a452bf5c7e362dfe25c53992a" Jul 24 04:07:02.715319 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:02.715288 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:07:02.715382 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:02.715363 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:07:02.715418 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:02.715402 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:07:03.721255 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:03.721221 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" event={"ID":"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57","Type":"ContainerStarted","Data":"1048fe3924d099a0d24908078f906c8a51b19fc731859c9f8b76634b74970abe"} Jul 24 04:07:03.990945 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:03.990764 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:07:04.121762 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.121724 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:07:04.122002 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.121976 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Jul 24 04:07:04.122061 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.122037 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" podUID="07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Jul 24 04:07:04.405597 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.405569 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw"] Jul 24 04:07:04.408497 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.408448 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.413791 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.413767 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-0cae5e-openshift-default-dockercfg-mtpf4\"" Jul 24 04:07:04.444978 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.444947 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw"] Jul 24 04:07:04.471559 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.471491 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:07:04.504081 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.503595 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-istio-data\") pod \"e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.504081 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.503645 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-istio-envoy\") pod \"e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.504081 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.503685 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-workload-certs\") pod \"e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.504081 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.503738 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6shc\" (UniqueName: \"kubernetes.io/projected/dc51c7c5-0c40-4f65-9da6-213029264e3b-kube-api-access-j6shc\") pod \"e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.504081 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.503767 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc51c7c5-0c40-4f65-9da6-213029264e3b-istiod-ca-cert\") pod \"e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.504081 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.503818 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-workload-socket\") pod \"e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.504081 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.503849 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-credential-socket\") pod \"e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.504081 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.503887 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc51c7c5-0c40-4f65-9da6-213029264e3b-istio-token\") pod \"e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.504081 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.503916 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc51c7c5-0c40-4f65-9da6-213029264e3b-istio-podinfo\") pod \"e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.604945 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.604895 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-workload-socket\") pod \"e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.604945 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.604952 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-credential-socket\") pod \"e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.605180 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.604990 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc51c7c5-0c40-4f65-9da6-213029264e3b-istio-token\") pod \"e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.605180 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.605016 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc51c7c5-0c40-4f65-9da6-213029264e3b-istio-podinfo\") pod \"e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.605180 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.605133 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-istio-data\") pod \"e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.605180 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.605164 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-istio-envoy\") pod \"e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.605398 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.605206 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-workload-certs\") pod \"e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.605398 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.605259 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6shc\" (UniqueName: \"kubernetes.io/projected/dc51c7c5-0c40-4f65-9da6-213029264e3b-kube-api-access-j6shc\") pod \"e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.605398 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.605287 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc51c7c5-0c40-4f65-9da6-213029264e3b-istiod-ca-cert\") pod \"e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.605398 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.605380 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-workload-socket\") pod \"e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.605634 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.605504 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-istio-data\") pod \"e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.605634 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.605597 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-credential-socket\") pod \"e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.605803 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.605783 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-workload-certs\") pod \"e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.606070 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.605972 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc51c7c5-0c40-4f65-9da6-213029264e3b-istiod-ca-cert\") pod \"e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.607626 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.607610 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-istio-envoy\") pod \"e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.607910 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.607892 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc51c7c5-0c40-4f65-9da6-213029264e3b-istio-podinfo\") pod \"e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.627057 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.627031 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc51c7c5-0c40-4f65-9da6-213029264e3b-istio-token\") pod \"e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.627187 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.627083 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6shc\" (UniqueName: \"kubernetes.io/projected/dc51c7c5-0c40-4f65-9da6-213029264e3b-kube-api-access-j6shc\") pod \"e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.719085 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.719004 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:04.742767 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.742727 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:07:04.899004 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.898915 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw"] Jul 24 04:07:04.900311 ip-10-0-134-147 kubenswrapper[2579]: W0724 04:07:04.900283 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc51c7c5_0c40_4f65_9da6_213029264e3b.slice/crio-20af7be2f70503d86b721732e055f8bf9bc8d0d6ad3edcbec07cef967fdcd4d0 WatchSource:0}: Error finding container 20af7be2f70503d86b721732e055f8bf9bc8d0d6ad3edcbec07cef967fdcd4d0: Status 404 returned error can't find the container with id 20af7be2f70503d86b721732e055f8bf9bc8d0d6ad3edcbec07cef967fdcd4d0 Jul 24 04:07:04.902692 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.902660 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:07:04.902783 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.902726 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:07:04.902783 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:04.902756 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:07:05.122227 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:05.122187 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Jul 24 04:07:05.122393 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:05.122266 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" podUID="07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Jul 24 04:07:05.735855 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:05.735816 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" event={"ID":"dc51c7c5-0c40-4f65-9da6-213029264e3b","Type":"ContainerStarted","Data":"798fbf495c02837af62f044ab506f58066bbfe08de065f6adef380ee81459cb8"} Jul 24 04:07:05.736036 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:05.735867 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" event={"ID":"dc51c7c5-0c40-4f65-9da6-213029264e3b","Type":"ContainerStarted","Data":"20af7be2f70503d86b721732e055f8bf9bc8d0d6ad3edcbec07cef967fdcd4d0"} Jul 24 04:07:05.759472 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:05.759420 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" podStartSLOduration=1.7594045189999998 podStartE2EDuration="1.759404519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 04:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 04:07:05.756527265 +0000 UTC m=+2975.195946726" watchObservedRunningTime="2026-07-24 04:07:05.759404519 +0000 UTC m=+2975.198824056" Jul 24 04:07:06.121964 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:06.121931 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Jul 24 04:07:06.122132 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:06.121984 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" podUID="07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Jul 24 04:07:06.720087 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:06.720049 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:06.725216 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:06.725192 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:06.741048 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:06.741016 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:06.741948 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:06.741928 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:07.122676 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:07.122643 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Jul 24 04:07:07.123068 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:07.122705 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" podUID="07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Jul 24 04:07:07.766524 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:07.766487 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:07:07.872909 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:07.872880 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:07:07.987563 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:07.987524 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:07:08.125827 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:08.125800 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:07:08.126276 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:08.126189 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:07:08.126930 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:08.126904 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:07:08.218317 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:08.218274 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:07:08.640905 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:08.640870 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:07:08.742789 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:08.742756 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:07:12.190786 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:12.190753 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:07:25.844800 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:25.844700 2579 generic.go:356] "Generic (PLEG): container finished" podID="dc51c7c5-0c40-4f65-9da6-213029264e3b" containerID="798fbf495c02837af62f044ab506f58066bbfe08de065f6adef380ee81459cb8" exitCode=0 Jul 24 04:07:25.844800 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:25.844752 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" event={"ID":"dc51c7c5-0c40-4f65-9da6-213029264e3b","Type":"ContainerDied","Data":"798fbf495c02837af62f044ab506f58066bbfe08de065f6adef380ee81459cb8"} Jul 24 04:07:25.845238 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:25.845137 2579 scope.go:117] "RemoveContainer" containerID="798fbf495c02837af62f044ab506f58066bbfe08de065f6adef380ee81459cb8" Jul 24 04:07:25.846213 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:25.846183 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:07:25.846274 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:25.846257 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:07:25.846310 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:25.846291 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:07:26.858203 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:26.858167 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" event={"ID":"dc51c7c5-0c40-4f65-9da6-213029264e3b","Type":"ContainerStarted","Data":"21c0c37831a0c4c09e42be6e583e674ad344831540f7b01cd3a20a76313aeb63"} Jul 24 04:07:27.720147 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:27.720113 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:27.720336 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:27.720316 2579 patch_prober.go:28] interesting pod/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Jul 24 04:07:27.720383 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:27.720367 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" podUID="dc51c7c5-0c40-4f65-9da6-213029264e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Jul 24 04:07:28.719676 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:28.719635 2579 patch_prober.go:28] interesting pod/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Jul 24 04:07:28.720107 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:28.719703 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" podUID="dc51c7c5-0c40-4f65-9da6-213029264e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Jul 24 04:07:29.720095 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:29.720058 2579 patch_prober.go:28] interesting pod/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Jul 24 04:07:29.720593 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:29.720132 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" podUID="dc51c7c5-0c40-4f65-9da6-213029264e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Jul 24 04:07:30.723225 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:30.723195 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:30.723639 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:30.723511 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:30.724362 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:30.724322 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:07:37.063745 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:07:37.063709 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:08:08.696845 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:08.696803 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:08:08.987451 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:08.987360 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t"] Jul 24 04:08:08.990220 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:08.990201 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:08.992643 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:08.992623 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-20e639-openshift-default-dockercfg-d7z95\"" Jul 24 04:08:09.002520 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.002497 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t"] Jul 24 04:08:09.008994 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.008969 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:08:09.080675 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.080638 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-istio-data\") pod \"e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:09.080943 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.080923 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-workload-certs\") pod \"e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:09.081142 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.081127 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-istio-envoy\") pod \"e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:09.081257 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.081242 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6fg9\" (UniqueName: \"kubernetes.io/projected/7cc0aea7-aecb-4e5f-9453-61feb2205530-kube-api-access-k6fg9\") pod \"e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:09.081380 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.081367 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc0aea7-aecb-4e5f-9453-61feb2205530-istio-token\") pod \"e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:09.101664 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.101637 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-credential-socket\") pod \"e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:09.102038 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.102019 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc0aea7-aecb-4e5f-9453-61feb2205530-istio-podinfo\") pod \"e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:09.102181 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.102166 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc0aea7-aecb-4e5f-9453-61feb2205530-istiod-ca-cert\") pod \"e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:09.102319 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.102305 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-workload-socket\") pod \"e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:09.203881 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.203847 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6fg9\" (UniqueName: \"kubernetes.io/projected/7cc0aea7-aecb-4e5f-9453-61feb2205530-kube-api-access-k6fg9\") pod \"e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:09.204091 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.203906 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc0aea7-aecb-4e5f-9453-61feb2205530-istio-token\") pod \"e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:09.204091 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.203936 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-credential-socket\") pod \"e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:09.204091 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.203997 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc0aea7-aecb-4e5f-9453-61feb2205530-istio-podinfo\") pod \"e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:09.204091 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.204023 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc0aea7-aecb-4e5f-9453-61feb2205530-istiod-ca-cert\") pod \"e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:09.204091 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.204076 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-workload-socket\") pod \"e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:09.204356 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.204104 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-istio-data\") pod \"e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:09.204356 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.204150 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-workload-certs\") pod \"e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:09.204356 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.204262 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-istio-envoy\") pod \"e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:09.204560 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.204426 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-credential-socket\") pod \"e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:09.204625 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.204609 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-workload-socket\") pod \"e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:09.204678 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.204651 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-workload-certs\") pod \"e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:09.204719 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.204708 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-istio-data\") pod \"e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:09.205233 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.205208 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc0aea7-aecb-4e5f-9453-61feb2205530-istiod-ca-cert\") pod \"e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:09.206685 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.206667 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-istio-envoy\") pod \"e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:09.206761 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.206718 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc0aea7-aecb-4e5f-9453-61feb2205530-istio-podinfo\") pod \"e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:09.211905 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.211885 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6fg9\" (UniqueName: \"kubernetes.io/projected/7cc0aea7-aecb-4e5f-9453-61feb2205530-kube-api-access-k6fg9\") pod \"e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:09.212013 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.211945 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc0aea7-aecb-4e5f-9453-61feb2205530-istio-token\") pod \"e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:09.223573 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.223532 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:08:09.302529 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.302412 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:09.453591 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.453534 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t"] Jul 24 04:08:09.456701 ip-10-0-134-147 kubenswrapper[2579]: W0724 04:08:09.456669 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc0aea7_aecb_4e5f_9453_61feb2205530.slice/crio-18a2b3d72e9fa9dfa35e7046cd47851274b4c5264ca88192e3777e979d599c98 WatchSource:0}: Error finding container 18a2b3d72e9fa9dfa35e7046cd47851274b4c5264ca88192e3777e979d599c98: Status 404 returned error can't find the container with id 18a2b3d72e9fa9dfa35e7046cd47851274b4c5264ca88192e3777e979d599c98 Jul 24 04:08:09.459099 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.459068 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:08:09.459185 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.459141 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:08:09.459185 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:09.459173 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:08:10.116449 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:10.116411 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" event={"ID":"7cc0aea7-aecb-4e5f-9453-61feb2205530","Type":"ContainerStarted","Data":"e788479871190d82d9527108ee1e5bbd25912d9645fdafcadf5756ed05e3769d"} Jul 24 04:08:10.116857 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:10.116492 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" event={"ID":"7cc0aea7-aecb-4e5f-9453-61feb2205530","Type":"ContainerStarted","Data":"18a2b3d72e9fa9dfa35e7046cd47851274b4c5264ca88192e3777e979d599c98"} Jul 24 04:08:10.138073 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:10.138017 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" podStartSLOduration=2.138003019 podStartE2EDuration="2.138003019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 04:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 04:08:10.136013242 +0000 UTC m=+3039.575432714" watchObservedRunningTime="2026-07-24 04:08:10.138003019 +0000 UTC m=+3039.577422552" Jul 24 04:08:10.302777 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:10.302742 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:10.307928 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:10.307903 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:11.121892 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:11.121851 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:11.122750 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:11.122731 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:12.071398 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:12.071364 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:08:12.169506 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:12.169479 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:08:12.269936 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:12.269900 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:08:12.907712 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:12.907671 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:08:13.011646 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:13.011598 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:08:13.092570 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:13.092540 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:08:16.570032 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:16.569999 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:08:20.951830 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:20.951797 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:08:26.468394 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:26.468319 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": read tcp 10.134.0.2:57314->10.134.0.75:15021: read: connection reset by peer" start-of-body= Jul 24 04:08:26.468797 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:26.468403 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" podUID="7cc0aea7-aecb-4e5f-9453-61feb2205530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": read tcp 10.134.0.2:57314->10.134.0.75:15021: read: connection reset by peer" Jul 24 04:08:27.233117 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:27.233084 2579 generic.go:356] "Generic (PLEG): container finished" podID="7cc0aea7-aecb-4e5f-9453-61feb2205530" containerID="e788479871190d82d9527108ee1e5bbd25912d9645fdafcadf5756ed05e3769d" exitCode=0 Jul 24 04:08:27.233291 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:27.233157 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" event={"ID":"7cc0aea7-aecb-4e5f-9453-61feb2205530","Type":"ContainerDied","Data":"e788479871190d82d9527108ee1e5bbd25912d9645fdafcadf5756ed05e3769d"} Jul 24 04:08:27.233630 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:27.233610 2579 scope.go:117] "RemoveContainer" containerID="e788479871190d82d9527108ee1e5bbd25912d9645fdafcadf5756ed05e3769d" Jul 24 04:08:27.234801 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:27.234773 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:08:27.234877 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:27.234864 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:08:27.234914 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:27.234898 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 24 04:08:28.241360 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:28.241328 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" event={"ID":"7cc0aea7-aecb-4e5f-9453-61feb2205530","Type":"ContainerStarted","Data":"d3d5659484804735332d8ab98fbddca762bf16413284da91db6256b76caea500"} Jul 24 04:08:28.303226 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:28.303192 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:28.303431 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:28.303389 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Jul 24 04:08:28.303534 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:28.303446 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" podUID="7cc0aea7-aecb-4e5f-9453-61feb2205530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Jul 24 04:08:29.303814 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:29.303783 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Jul 24 04:08:29.304202 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:29.303837 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" podUID="7cc0aea7-aecb-4e5f-9453-61feb2205530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Jul 24 04:08:30.303098 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:30.303058 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Jul 24 04:08:30.303280 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:30.303129 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" podUID="7cc0aea7-aecb-4e5f-9453-61feb2205530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Jul 24 04:08:31.303562 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:31.303526 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Jul 24 04:08:31.303950 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:31.303597 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" podUID="7cc0aea7-aecb-4e5f-9453-61feb2205530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Jul 24 04:08:31.758625 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:31.758590 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:08:31.814861 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:31.814826 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:08:32.307061 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:32.307026 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:32.307522 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:32.307488 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:32.308151 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:32.308134 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:38.059188 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:38.059132 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:08:38.163062 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:38.163026 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:08:38.260122 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:38.260085 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:08:42.555480 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:42.555430 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:08:42.939607 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:42.939569 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:08:48.677005 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:48.676966 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:08:48.777009 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:48.776976 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:08:48.926478 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:48.926425 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t"] Jul 24 04:08:48.926707 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:48.926675 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" podUID="7cc0aea7-aecb-4e5f-9453-61feb2205530" containerName="istio-proxy" containerID="cri-o://d3d5659484804735332d8ab98fbddca762bf16413284da91db6256b76caea500" gracePeriod=30 Jul 24 04:08:48.943580 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:48.943555 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:08:49.189089 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:49.188985 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:08:54.190369 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.190307 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:54.366291 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.366258 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc0aea7-aecb-4e5f-9453-61feb2205530-istio-token\") pod \"7cc0aea7-aecb-4e5f-9453-61feb2205530\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " Jul 24 04:08:54.366537 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.366328 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6fg9\" (UniqueName: \"kubernetes.io/projected/7cc0aea7-aecb-4e5f-9453-61feb2205530-kube-api-access-k6fg9\") pod \"7cc0aea7-aecb-4e5f-9453-61feb2205530\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " Jul 24 04:08:54.366537 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.366387 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-workload-certs\") pod \"7cc0aea7-aecb-4e5f-9453-61feb2205530\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " Jul 24 04:08:54.366537 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.366417 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-istio-data\") pod \"7cc0aea7-aecb-4e5f-9453-61feb2205530\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " Jul 24 04:08:54.366537 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.366488 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-istio-envoy\") pod \"7cc0aea7-aecb-4e5f-9453-61feb2205530\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " Jul 24 04:08:54.366537 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.366513 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc0aea7-aecb-4e5f-9453-61feb2205530-istio-podinfo\") pod \"7cc0aea7-aecb-4e5f-9453-61feb2205530\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " Jul 24 04:08:54.366824 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.366541 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-workload-socket\") pod \"7cc0aea7-aecb-4e5f-9453-61feb2205530\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " Jul 24 04:08:54.366824 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.366584 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc0aea7-aecb-4e5f-9453-61feb2205530-istiod-ca-cert\") pod \"7cc0aea7-aecb-4e5f-9453-61feb2205530\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " Jul 24 04:08:54.366824 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.366612 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-credential-socket\") pod \"7cc0aea7-aecb-4e5f-9453-61feb2205530\" (UID: \"7cc0aea7-aecb-4e5f-9453-61feb2205530\") " Jul 24 04:08:54.366997 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.366823 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7cc0aea7-aecb-4e5f-9453-61feb2205530" (UID: "7cc0aea7-aecb-4e5f-9453-61feb2205530"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:08:54.366997 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.366932 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7cc0aea7-aecb-4e5f-9453-61feb2205530" (UID: "7cc0aea7-aecb-4e5f-9453-61feb2205530"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:08:54.367113 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.367057 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7cc0aea7-aecb-4e5f-9453-61feb2205530" (UID: "7cc0aea7-aecb-4e5f-9453-61feb2205530"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:08:54.367192 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.367174 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-workload-certs\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:08:54.367257 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.367197 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-workload-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:08:54.367257 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.367211 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-credential-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:08:54.367433 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.367405 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cc0aea7-aecb-4e5f-9453-61feb2205530-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7cc0aea7-aecb-4e5f-9453-61feb2205530" (UID: "7cc0aea7-aecb-4e5f-9453-61feb2205530"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 04:08:54.367841 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.367814 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-istio-data" (OuterVolumeSpecName: "istio-data") pod "7cc0aea7-aecb-4e5f-9453-61feb2205530" (UID: "7cc0aea7-aecb-4e5f-9453-61feb2205530"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:08:54.368964 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.368932 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cc0aea7-aecb-4e5f-9453-61feb2205530-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7cc0aea7-aecb-4e5f-9453-61feb2205530" (UID: "7cc0aea7-aecb-4e5f-9453-61feb2205530"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 04:08:54.369157 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.369139 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7cc0aea7-aecb-4e5f-9453-61feb2205530" (UID: "7cc0aea7-aecb-4e5f-9453-61feb2205530"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:08:54.369208 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.369147 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc0aea7-aecb-4e5f-9453-61feb2205530-kube-api-access-k6fg9" (OuterVolumeSpecName: "kube-api-access-k6fg9") pod "7cc0aea7-aecb-4e5f-9453-61feb2205530" (UID: "7cc0aea7-aecb-4e5f-9453-61feb2205530"). InnerVolumeSpecName "kube-api-access-k6fg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 04:08:54.369243 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.369208 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc0aea7-aecb-4e5f-9453-61feb2205530-istio-token" (OuterVolumeSpecName: "istio-token") pod "7cc0aea7-aecb-4e5f-9453-61feb2205530" (UID: "7cc0aea7-aecb-4e5f-9453-61feb2205530"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 04:08:54.387343 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.387317 2579 generic.go:356] "Generic (PLEG): container finished" podID="7cc0aea7-aecb-4e5f-9453-61feb2205530" containerID="d3d5659484804735332d8ab98fbddca762bf16413284da91db6256b76caea500" exitCode=0 Jul 24 04:08:54.387442 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.387371 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" Jul 24 04:08:54.387442 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.387383 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" event={"ID":"7cc0aea7-aecb-4e5f-9453-61feb2205530","Type":"ContainerDied","Data":"d3d5659484804735332d8ab98fbddca762bf16413284da91db6256b76caea500"} Jul 24 04:08:54.387442 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.387435 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t" event={"ID":"7cc0aea7-aecb-4e5f-9453-61feb2205530","Type":"ContainerDied","Data":"18a2b3d72e9fa9dfa35e7046cd47851274b4c5264ca88192e3777e979d599c98"} Jul 24 04:08:54.387637 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.387477 2579 scope.go:117] "RemoveContainer" containerID="d3d5659484804735332d8ab98fbddca762bf16413284da91db6256b76caea500" Jul 24 04:08:54.397446 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.397426 2579 scope.go:117] "RemoveContainer" containerID="e788479871190d82d9527108ee1e5bbd25912d9645fdafcadf5756ed05e3769d" Jul 24 04:08:54.407940 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.407919 2579 scope.go:117] "RemoveContainer" containerID="d3d5659484804735332d8ab98fbddca762bf16413284da91db6256b76caea500" Jul 24 04:08:54.408244 ip-10-0-134-147 kubenswrapper[2579]: E0724 04:08:54.408227 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3d5659484804735332d8ab98fbddca762bf16413284da91db6256b76caea500\": container with ID starting with d3d5659484804735332d8ab98fbddca762bf16413284da91db6256b76caea500 not found: ID does not exist" containerID="d3d5659484804735332d8ab98fbddca762bf16413284da91db6256b76caea500" Jul 24 04:08:54.408321 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.408253 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3d5659484804735332d8ab98fbddca762bf16413284da91db6256b76caea500"} err="failed to get container status \"d3d5659484804735332d8ab98fbddca762bf16413284da91db6256b76caea500\": rpc error: code = NotFound desc = could not find container \"d3d5659484804735332d8ab98fbddca762bf16413284da91db6256b76caea500\": container with ID starting with d3d5659484804735332d8ab98fbddca762bf16413284da91db6256b76caea500 not found: ID does not exist" Jul 24 04:08:54.408321 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.408272 2579 scope.go:117] "RemoveContainer" containerID="e788479871190d82d9527108ee1e5bbd25912d9645fdafcadf5756ed05e3769d" Jul 24 04:08:54.408578 ip-10-0-134-147 kubenswrapper[2579]: E0724 04:08:54.408550 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e788479871190d82d9527108ee1e5bbd25912d9645fdafcadf5756ed05e3769d\": container with ID starting with e788479871190d82d9527108ee1e5bbd25912d9645fdafcadf5756ed05e3769d not found: ID does not exist" containerID="e788479871190d82d9527108ee1e5bbd25912d9645fdafcadf5756ed05e3769d" Jul 24 04:08:54.408667 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.408588 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e788479871190d82d9527108ee1e5bbd25912d9645fdafcadf5756ed05e3769d"} err="failed to get container status \"e788479871190d82d9527108ee1e5bbd25912d9645fdafcadf5756ed05e3769d\": rpc error: code = NotFound desc = could not find container \"e788479871190d82d9527108ee1e5bbd25912d9645fdafcadf5756ed05e3769d\": container with ID starting with e788479871190d82d9527108ee1e5bbd25912d9645fdafcadf5756ed05e3769d not found: ID does not exist" Jul 24 04:08:54.410137 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.410115 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t"] Jul 24 04:08:54.416263 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.416242 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-20e639-openshift-default-5b777758c8-l687t"] Jul 24 04:08:54.468606 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.468533 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-istio-data\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:08:54.468606 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.468559 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc0aea7-aecb-4e5f-9453-61feb2205530-istio-envoy\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:08:54.468606 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.468568 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc0aea7-aecb-4e5f-9453-61feb2205530-istio-podinfo\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:08:54.468606 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.468576 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc0aea7-aecb-4e5f-9453-61feb2205530-istiod-ca-cert\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:08:54.468606 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.468586 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc0aea7-aecb-4e5f-9453-61feb2205530-istio-token\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:08:54.468606 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:54.468594 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6fg9\" (UniqueName: \"kubernetes.io/projected/7cc0aea7-aecb-4e5f-9453-61feb2205530-kube-api-access-k6fg9\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:08:55.034388 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:08:55.034353 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cc0aea7-aecb-4e5f-9453-61feb2205530" path="/var/lib/kubelet/pods/7cc0aea7-aecb-4e5f-9453-61feb2205530/volumes" Jul 24 04:09:08.302669 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:09:08.302632 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:09:09.690426 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:09:09.690384 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:09:12.817403 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:09:12.817367 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:09:13.202612 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:09:13.202581 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:09:13.302726 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:09:13.302695 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:09:18.284593 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:09:18.284561 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:09:46.442716 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:09:46.442675 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:09:46.540232 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:09:46.540200 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:09:55.247318 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:09:55.247238 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:09:55.375309 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:09:55.375277 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:10:01.047016 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:01.046981 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:10:01.146879 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:01.146845 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:10:01.294836 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:01.294797 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp"] Jul 24 04:10:01.295116 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:01.295080 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" podUID="07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57" containerName="istio-proxy" containerID="cri-o://1048fe3924d099a0d24908078f906c8a51b19fc731859c9f8b76634b74970abe" gracePeriod=30 Jul 24 04:10:01.320104 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:01.320007 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:10:01.614867 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:01.614823 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:10:06.569134 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.569072 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:10:06.687760 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.687732 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istio-data\") pod \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " Jul 24 04:10:06.687942 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.687773 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntq6k\" (UniqueName: \"kubernetes.io/projected/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-kube-api-access-ntq6k\") pod \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " Jul 24 04:10:06.687942 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.687793 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-workload-socket\") pod \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " Jul 24 04:10:06.687942 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.687819 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-workload-certs\") pod \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " Jul 24 04:10:06.688109 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.688020 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istiod-ca-cert\") pod \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " Jul 24 04:10:06.688109 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.688039 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57" (UID: "07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:10:06.688109 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.688098 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-credential-socket\") pod \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " Jul 24 04:10:06.688262 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.688146 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istio-token\") pod \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " Jul 24 04:10:06.688262 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.688098 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57" (UID: "07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:10:06.688262 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.688239 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istio-podinfo\") pod \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " Jul 24 04:10:06.688408 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.688317 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istio-envoy\") pod \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\" (UID: \"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57\") " Jul 24 04:10:06.688408 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.688340 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57" (UID: "07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:10:06.688408 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.688358 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57" (UID: "07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 04:10:06.688711 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.688679 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-workload-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:10:06.688828 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.688723 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-workload-certs\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:10:06.688828 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.688738 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istiod-ca-cert\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:10:06.688828 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.688750 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-credential-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:10:06.688828 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.688682 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istio-data" (OuterVolumeSpecName: "istio-data") pod "07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57" (UID: "07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:10:06.690523 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.690488 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istio-token" (OuterVolumeSpecName: "istio-token") pod "07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57" (UID: "07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 04:10:06.690633 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.690570 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57" (UID: "07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:10:06.690755 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.690734 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-kube-api-access-ntq6k" (OuterVolumeSpecName: "kube-api-access-ntq6k") pod "07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57" (UID: "07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57"). InnerVolumeSpecName "kube-api-access-ntq6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 04:10:06.690833 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.690814 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57" (UID: "07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 04:10:06.785454 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.785414 2579 generic.go:356] "Generic (PLEG): container finished" podID="07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57" containerID="1048fe3924d099a0d24908078f906c8a51b19fc731859c9f8b76634b74970abe" exitCode=0 Jul 24 04:10:06.785621 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.785484 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" Jul 24 04:10:06.785621 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.785498 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" event={"ID":"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57","Type":"ContainerDied","Data":"1048fe3924d099a0d24908078f906c8a51b19fc731859c9f8b76634b74970abe"} Jul 24 04:10:06.785621 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.785555 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp" event={"ID":"07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57","Type":"ContainerDied","Data":"7cb6e4696b711de256aec5767940851d38fc7951f1bba90b2d953a4659bd4190"} Jul 24 04:10:06.785621 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.785576 2579 scope.go:117] "RemoveContainer" containerID="1048fe3924d099a0d24908078f906c8a51b19fc731859c9f8b76634b74970abe" Jul 24 04:10:06.789373 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.789345 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istio-envoy\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:10:06.789497 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.789378 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istio-data\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:10:06.789497 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.789391 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntq6k\" (UniqueName: \"kubernetes.io/projected/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-kube-api-access-ntq6k\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:10:06.789497 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.789400 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istio-token\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:10:06.789497 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.789411 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57-istio-podinfo\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:10:06.795129 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.795111 2579 scope.go:117] "RemoveContainer" containerID="5162645ce5442392652579e965b52ce18098996a452bf5c7e362dfe25c53992a" Jul 24 04:10:06.803693 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.803659 2579 scope.go:117] "RemoveContainer" containerID="1048fe3924d099a0d24908078f906c8a51b19fc731859c9f8b76634b74970abe" Jul 24 04:10:06.803992 ip-10-0-134-147 kubenswrapper[2579]: E0724 04:10:06.803951 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1048fe3924d099a0d24908078f906c8a51b19fc731859c9f8b76634b74970abe\": container with ID starting with 1048fe3924d099a0d24908078f906c8a51b19fc731859c9f8b76634b74970abe not found: ID does not exist" containerID="1048fe3924d099a0d24908078f906c8a51b19fc731859c9f8b76634b74970abe" Jul 24 04:10:06.804077 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.803996 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1048fe3924d099a0d24908078f906c8a51b19fc731859c9f8b76634b74970abe"} err="failed to get container status \"1048fe3924d099a0d24908078f906c8a51b19fc731859c9f8b76634b74970abe\": rpc error: code = NotFound desc = could not find container \"1048fe3924d099a0d24908078f906c8a51b19fc731859c9f8b76634b74970abe\": container with ID starting with 1048fe3924d099a0d24908078f906c8a51b19fc731859c9f8b76634b74970abe not found: ID does not exist" Jul 24 04:10:06.804077 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.804018 2579 scope.go:117] "RemoveContainer" containerID="5162645ce5442392652579e965b52ce18098996a452bf5c7e362dfe25c53992a" Jul 24 04:10:06.804444 ip-10-0-134-147 kubenswrapper[2579]: E0724 04:10:06.804380 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5162645ce5442392652579e965b52ce18098996a452bf5c7e362dfe25c53992a\": container with ID starting with 5162645ce5442392652579e965b52ce18098996a452bf5c7e362dfe25c53992a not found: ID does not exist" containerID="5162645ce5442392652579e965b52ce18098996a452bf5c7e362dfe25c53992a" Jul 24 04:10:06.804444 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.804414 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5162645ce5442392652579e965b52ce18098996a452bf5c7e362dfe25c53992a"} err="failed to get container status \"5162645ce5442392652579e965b52ce18098996a452bf5c7e362dfe25c53992a\": rpc error: code = NotFound desc = could not find container \"5162645ce5442392652579e965b52ce18098996a452bf5c7e362dfe25c53992a\": container with ID starting with 5162645ce5442392652579e965b52ce18098996a452bf5c7e362dfe25c53992a not found: ID does not exist" Jul 24 04:10:06.806849 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.806829 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp"] Jul 24 04:10:06.810521 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:06.810503 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-733dde-openshift-default-76b94c4df8-qz4cp"] Jul 24 04:10:07.034299 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:07.034264 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57" path="/var/lib/kubelet/pods/07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57/volumes" Jul 24 04:10:12.199271 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:12.199236 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:10:12.308953 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:12.308923 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:10:12.409405 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:12.409370 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:10:16.734730 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:16.734689 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:10:17.118531 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:17.118498 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:10:22.871054 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:22.871016 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:10:22.931506 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:22.931450 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:10:23.109193 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:23.109161 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw"] Jul 24 04:10:23.109448 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:23.109401 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" podUID="dc51c7c5-0c40-4f65-9da6-213029264e3b" containerName="istio-proxy" containerID="cri-o://21c0c37831a0c4c09e42be6e583e674ad344831540f7b01cd3a20a76313aeb63" gracePeriod=30 Jul 24 04:10:23.131280 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:23.131219 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:10:23.369897 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:23.369869 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:10:28.364488 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.364446 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:10:28.514654 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.514554 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-workload-certs\") pod \"dc51c7c5-0c40-4f65-9da6-213029264e3b\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " Jul 24 04:10:28.514654 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.514606 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-istio-envoy\") pod \"dc51c7c5-0c40-4f65-9da6-213029264e3b\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " Jul 24 04:10:28.514654 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.514644 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc51c7c5-0c40-4f65-9da6-213029264e3b-istio-podinfo\") pod \"dc51c7c5-0c40-4f65-9da6-213029264e3b\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " Jul 24 04:10:28.514940 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.514698 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6shc\" (UniqueName: \"kubernetes.io/projected/dc51c7c5-0c40-4f65-9da6-213029264e3b-kube-api-access-j6shc\") pod \"dc51c7c5-0c40-4f65-9da6-213029264e3b\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " Jul 24 04:10:28.514940 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.514721 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-istio-data\") pod \"dc51c7c5-0c40-4f65-9da6-213029264e3b\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " Jul 24 04:10:28.514940 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.514767 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-credential-socket\") pod \"dc51c7c5-0c40-4f65-9da6-213029264e3b\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " Jul 24 04:10:28.514940 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.514853 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-workload-socket\") pod \"dc51c7c5-0c40-4f65-9da6-213029264e3b\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " Jul 24 04:10:28.514940 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.514889 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc51c7c5-0c40-4f65-9da6-213029264e3b-istiod-ca-cert\") pod \"dc51c7c5-0c40-4f65-9da6-213029264e3b\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " Jul 24 04:10:28.515184 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.514949 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc51c7c5-0c40-4f65-9da6-213029264e3b-istio-token\") pod \"dc51c7c5-0c40-4f65-9da6-213029264e3b\" (UID: \"dc51c7c5-0c40-4f65-9da6-213029264e3b\") " Jul 24 04:10:28.515184 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.514849 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dc51c7c5-0c40-4f65-9da6-213029264e3b" (UID: "dc51c7c5-0c40-4f65-9da6-213029264e3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:10:28.515184 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.515071 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dc51c7c5-0c40-4f65-9da6-213029264e3b" (UID: "dc51c7c5-0c40-4f65-9da6-213029264e3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:10:28.515184 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.515100 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dc51c7c5-0c40-4f65-9da6-213029264e3b" (UID: "dc51c7c5-0c40-4f65-9da6-213029264e3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:10:28.515417 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.515389 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc51c7c5-0c40-4f65-9da6-213029264e3b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dc51c7c5-0c40-4f65-9da6-213029264e3b" (UID: "dc51c7c5-0c40-4f65-9da6-213029264e3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 04:10:28.515616 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.515594 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-workload-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:10:28.515874 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.515621 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-workload-certs\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:10:28.515874 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.515636 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-credential-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:10:28.515874 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.515848 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-istio-data" (OuterVolumeSpecName: "istio-data") pod "dc51c7c5-0c40-4f65-9da6-213029264e3b" (UID: "dc51c7c5-0c40-4f65-9da6-213029264e3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:10:28.517055 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.517028 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dc51c7c5-0c40-4f65-9da6-213029264e3b" (UID: "dc51c7c5-0c40-4f65-9da6-213029264e3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:10:28.517055 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.517040 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc51c7c5-0c40-4f65-9da6-213029264e3b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dc51c7c5-0c40-4f65-9da6-213029264e3b" (UID: "dc51c7c5-0c40-4f65-9da6-213029264e3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 04:10:28.517190 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.517116 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc51c7c5-0c40-4f65-9da6-213029264e3b-kube-api-access-j6shc" (OuterVolumeSpecName: "kube-api-access-j6shc") pod "dc51c7c5-0c40-4f65-9da6-213029264e3b" (UID: "dc51c7c5-0c40-4f65-9da6-213029264e3b"). InnerVolumeSpecName "kube-api-access-j6shc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 04:10:28.517235 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.517190 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc51c7c5-0c40-4f65-9da6-213029264e3b-istio-token" (OuterVolumeSpecName: "istio-token") pod "dc51c7c5-0c40-4f65-9da6-213029264e3b" (UID: "dc51c7c5-0c40-4f65-9da6-213029264e3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 04:10:28.616128 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.616097 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc51c7c5-0c40-4f65-9da6-213029264e3b-istiod-ca-cert\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:10:28.616128 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.616123 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc51c7c5-0c40-4f65-9da6-213029264e3b-istio-token\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:10:28.616128 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.616132 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-istio-envoy\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:10:28.616348 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.616140 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc51c7c5-0c40-4f65-9da6-213029264e3b-istio-podinfo\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:10:28.616348 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.616148 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6shc\" (UniqueName: \"kubernetes.io/projected/dc51c7c5-0c40-4f65-9da6-213029264e3b-kube-api-access-j6shc\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:10:28.616348 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.616156 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc51c7c5-0c40-4f65-9da6-213029264e3b-istio-data\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:10:28.907513 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.907437 2579 generic.go:356] "Generic (PLEG): container finished" podID="dc51c7c5-0c40-4f65-9da6-213029264e3b" containerID="21c0c37831a0c4c09e42be6e583e674ad344831540f7b01cd3a20a76313aeb63" exitCode=0 Jul 24 04:10:28.907513 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.907495 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" event={"ID":"dc51c7c5-0c40-4f65-9da6-213029264e3b","Type":"ContainerDied","Data":"21c0c37831a0c4c09e42be6e583e674ad344831540f7b01cd3a20a76313aeb63"} Jul 24 04:10:28.907722 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.907519 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" Jul 24 04:10:28.907722 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.907559 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw" event={"ID":"dc51c7c5-0c40-4f65-9da6-213029264e3b","Type":"ContainerDied","Data":"20af7be2f70503d86b721732e055f8bf9bc8d0d6ad3edcbec07cef967fdcd4d0"} Jul 24 04:10:28.907722 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.907585 2579 scope.go:117] "RemoveContainer" containerID="21c0c37831a0c4c09e42be6e583e674ad344831540f7b01cd3a20a76313aeb63" Jul 24 04:10:28.918146 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.918126 2579 scope.go:117] "RemoveContainer" containerID="798fbf495c02837af62f044ab506f58066bbfe08de065f6adef380ee81459cb8" Jul 24 04:10:28.926233 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.926216 2579 scope.go:117] "RemoveContainer" containerID="21c0c37831a0c4c09e42be6e583e674ad344831540f7b01cd3a20a76313aeb63" Jul 24 04:10:28.926494 ip-10-0-134-147 kubenswrapper[2579]: E0724 04:10:28.926453 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21c0c37831a0c4c09e42be6e583e674ad344831540f7b01cd3a20a76313aeb63\": container with ID starting with 21c0c37831a0c4c09e42be6e583e674ad344831540f7b01cd3a20a76313aeb63 not found: ID does not exist" containerID="21c0c37831a0c4c09e42be6e583e674ad344831540f7b01cd3a20a76313aeb63" Jul 24 04:10:28.926552 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.926502 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21c0c37831a0c4c09e42be6e583e674ad344831540f7b01cd3a20a76313aeb63"} err="failed to get container status \"21c0c37831a0c4c09e42be6e583e674ad344831540f7b01cd3a20a76313aeb63\": rpc error: code = NotFound desc = could not find container \"21c0c37831a0c4c09e42be6e583e674ad344831540f7b01cd3a20a76313aeb63\": container with ID starting with 21c0c37831a0c4c09e42be6e583e674ad344831540f7b01cd3a20a76313aeb63 not found: ID does not exist" Jul 24 04:10:28.926552 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.926519 2579 scope.go:117] "RemoveContainer" containerID="798fbf495c02837af62f044ab506f58066bbfe08de065f6adef380ee81459cb8" Jul 24 04:10:28.926840 ip-10-0-134-147 kubenswrapper[2579]: E0724 04:10:28.926820 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"798fbf495c02837af62f044ab506f58066bbfe08de065f6adef380ee81459cb8\": container with ID starting with 798fbf495c02837af62f044ab506f58066bbfe08de065f6adef380ee81459cb8 not found: ID does not exist" containerID="798fbf495c02837af62f044ab506f58066bbfe08de065f6adef380ee81459cb8" Jul 24 04:10:28.926938 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.926847 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"798fbf495c02837af62f044ab506f58066bbfe08de065f6adef380ee81459cb8"} err="failed to get container status \"798fbf495c02837af62f044ab506f58066bbfe08de065f6adef380ee81459cb8\": rpc error: code = NotFound desc = could not find container \"798fbf495c02837af62f044ab506f58066bbfe08de065f6adef380ee81459cb8\": container with ID starting with 798fbf495c02837af62f044ab506f58066bbfe08de065f6adef380ee81459cb8 not found: ID does not exist" Jul 24 04:10:28.929745 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.929725 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw"] Jul 24 04:10:28.933263 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:28.933244 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-0cae5e-openshift-default-6bcd8d548b-d9ntw"] Jul 24 04:10:29.034263 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:29.034230 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc51c7c5-0c40-4f65-9da6-213029264e3b" path="/var/lib/kubelet/pods/dc51c7c5-0c40-4f65-9da6-213029264e3b/volumes" Jul 24 04:10:33.942210 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:33.942179 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:10:34.005935 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:34.005904 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:10:34.065070 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:34.065031 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:10:38.627155 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:38.627110 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:10:39.016762 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:39.016673 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:10:45.244031 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:45.243981 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:10:45.348240 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:45.348197 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:10:45.519274 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:45.519185 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt"] Jul 24 04:10:45.519451 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:45.519424 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" podUID="39ad2308-505c-4c87-9e7b-6a6508b52375" containerName="istio-proxy" containerID="cri-o://dfd1b00a60c3fcc212d847bf7e3ee75e8d1c12c12337d7926bad3a298bf69f68" gracePeriod=30 Jul 24 04:10:45.540311 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:45.540285 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:10:45.759301 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:45.759270 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:10:46.923456 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:46.923419 2579 patch_prober.go:28] interesting pod/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 04:10:46.923923 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:46.923509 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" podUID="39ad2308-505c-4c87-9e7b-6a6508b52375" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 04:10:50.802860 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:50.802835 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:10:50.827817 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:50.827791 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-istio-envoy\") pod \"39ad2308-505c-4c87-9e7b-6a6508b52375\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " Jul 24 04:10:50.827988 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:50.827873 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39ad2308-505c-4c87-9e7b-6a6508b52375-istio-token\") pod \"39ad2308-505c-4c87-9e7b-6a6508b52375\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " Jul 24 04:10:50.827988 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:50.827898 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-istio-data\") pod \"39ad2308-505c-4c87-9e7b-6a6508b52375\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " Jul 24 04:10:50.827988 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:50.827977 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ad2308-505c-4c87-9e7b-6a6508b52375-istiod-ca-cert\") pod \"39ad2308-505c-4c87-9e7b-6a6508b52375\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " Jul 24 04:10:50.828157 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:50.828033 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ad2308-505c-4c87-9e7b-6a6508b52375-istio-podinfo\") pod \"39ad2308-505c-4c87-9e7b-6a6508b52375\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " Jul 24 04:10:50.828157 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:50.828061 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-workload-socket\") pod \"39ad2308-505c-4c87-9e7b-6a6508b52375\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " Jul 24 04:10:50.828157 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:50.828102 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-workload-certs\") pod \"39ad2308-505c-4c87-9e7b-6a6508b52375\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " Jul 24 04:10:50.828157 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:50.828129 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzc9z\" (UniqueName: \"kubernetes.io/projected/39ad2308-505c-4c87-9e7b-6a6508b52375-kube-api-access-bzc9z\") pod \"39ad2308-505c-4c87-9e7b-6a6508b52375\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " Jul 24 04:10:50.828351 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:50.828165 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-credential-socket\") pod \"39ad2308-505c-4c87-9e7b-6a6508b52375\" (UID: \"39ad2308-505c-4c87-9e7b-6a6508b52375\") " Jul 24 04:10:50.829047 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:50.828506 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "39ad2308-505c-4c87-9e7b-6a6508b52375" (UID: "39ad2308-505c-4c87-9e7b-6a6508b52375"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:10:50.829047 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:50.828734 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "39ad2308-505c-4c87-9e7b-6a6508b52375" (UID: "39ad2308-505c-4c87-9e7b-6a6508b52375"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:10:50.829047 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:50.829016 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39ad2308-505c-4c87-9e7b-6a6508b52375-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "39ad2308-505c-4c87-9e7b-6a6508b52375" (UID: "39ad2308-505c-4c87-9e7b-6a6508b52375"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 04:10:50.829047 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:50.829045 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "39ad2308-505c-4c87-9e7b-6a6508b52375" (UID: "39ad2308-505c-4c87-9e7b-6a6508b52375"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:10:50.829524 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:50.829499 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-istio-data" (OuterVolumeSpecName: "istio-data") pod "39ad2308-505c-4c87-9e7b-6a6508b52375" (UID: "39ad2308-505c-4c87-9e7b-6a6508b52375"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:10:50.831092 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:50.831050 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ad2308-505c-4c87-9e7b-6a6508b52375-istio-token" (OuterVolumeSpecName: "istio-token") pod "39ad2308-505c-4c87-9e7b-6a6508b52375" (UID: "39ad2308-505c-4c87-9e7b-6a6508b52375"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 04:10:50.831092 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:50.831070 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "39ad2308-505c-4c87-9e7b-6a6508b52375" (UID: "39ad2308-505c-4c87-9e7b-6a6508b52375"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:10:50.831591 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:50.831566 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ad2308-505c-4c87-9e7b-6a6508b52375-kube-api-access-bzc9z" (OuterVolumeSpecName: "kube-api-access-bzc9z") pod "39ad2308-505c-4c87-9e7b-6a6508b52375" (UID: "39ad2308-505c-4c87-9e7b-6a6508b52375"). InnerVolumeSpecName "kube-api-access-bzc9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 04:10:50.831684 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:50.831660 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39ad2308-505c-4c87-9e7b-6a6508b52375-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "39ad2308-505c-4c87-9e7b-6a6508b52375" (UID: "39ad2308-505c-4c87-9e7b-6a6508b52375"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 04:10:50.929536 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:50.929504 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39ad2308-505c-4c87-9e7b-6a6508b52375-istio-token\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:10:50.929536 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:50.929531 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-istio-data\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:10:50.929536 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:50.929543 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ad2308-505c-4c87-9e7b-6a6508b52375-istiod-ca-cert\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:10:50.929788 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:50.929551 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ad2308-505c-4c87-9e7b-6a6508b52375-istio-podinfo\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:10:50.929788 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:50.929559 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-workload-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:10:50.929788 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:50.929567 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-workload-certs\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:10:50.929788 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:50.929576 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzc9z\" (UniqueName: \"kubernetes.io/projected/39ad2308-505c-4c87-9e7b-6a6508b52375-kube-api-access-bzc9z\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:10:50.929788 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:50.929583 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-credential-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:10:50.929788 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:50.929591 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ad2308-505c-4c87-9e7b-6a6508b52375-istio-envoy\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:10:51.027619 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:51.027569 2579 generic.go:356] "Generic (PLEG): container finished" podID="39ad2308-505c-4c87-9e7b-6a6508b52375" containerID="dfd1b00a60c3fcc212d847bf7e3ee75e8d1c12c12337d7926bad3a298bf69f68" exitCode=0 Jul 24 04:10:51.032855 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:51.032776 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" Jul 24 04:10:51.038785 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:51.038757 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" event={"ID":"39ad2308-505c-4c87-9e7b-6a6508b52375","Type":"ContainerDied","Data":"dfd1b00a60c3fcc212d847bf7e3ee75e8d1c12c12337d7926bad3a298bf69f68"} Jul 24 04:10:51.038925 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:51.038799 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt" event={"ID":"39ad2308-505c-4c87-9e7b-6a6508b52375","Type":"ContainerDied","Data":"9c8845c351d4129008c40154c330b4d428866764c23a283e2267d8a715ea994d"} Jul 24 04:10:51.038925 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:51.038821 2579 scope.go:117] "RemoveContainer" containerID="dfd1b00a60c3fcc212d847bf7e3ee75e8d1c12c12337d7926bad3a298bf69f68" Jul 24 04:10:51.048235 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:51.048214 2579 scope.go:117] "RemoveContainer" containerID="87e19458f5bbdae659ce77616dc0522e0eef706d9a5ef7b3ba2f7ed93a327c48" Jul 24 04:10:51.056179 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:51.056163 2579 scope.go:117] "RemoveContainer" containerID="dfd1b00a60c3fcc212d847bf7e3ee75e8d1c12c12337d7926bad3a298bf69f68" Jul 24 04:10:51.056427 ip-10-0-134-147 kubenswrapper[2579]: E0724 04:10:51.056411 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfd1b00a60c3fcc212d847bf7e3ee75e8d1c12c12337d7926bad3a298bf69f68\": container with ID starting with dfd1b00a60c3fcc212d847bf7e3ee75e8d1c12c12337d7926bad3a298bf69f68 not found: ID does not exist" containerID="dfd1b00a60c3fcc212d847bf7e3ee75e8d1c12c12337d7926bad3a298bf69f68" Jul 24 04:10:51.056499 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:51.056435 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfd1b00a60c3fcc212d847bf7e3ee75e8d1c12c12337d7926bad3a298bf69f68"} err="failed to get container status \"dfd1b00a60c3fcc212d847bf7e3ee75e8d1c12c12337d7926bad3a298bf69f68\": rpc error: code = NotFound desc = could not find container \"dfd1b00a60c3fcc212d847bf7e3ee75e8d1c12c12337d7926bad3a298bf69f68\": container with ID starting with dfd1b00a60c3fcc212d847bf7e3ee75e8d1c12c12337d7926bad3a298bf69f68 not found: ID does not exist" Jul 24 04:10:51.056499 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:51.056450 2579 scope.go:117] "RemoveContainer" containerID="87e19458f5bbdae659ce77616dc0522e0eef706d9a5ef7b3ba2f7ed93a327c48" Jul 24 04:10:51.056680 ip-10-0-134-147 kubenswrapper[2579]: E0724 04:10:51.056666 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87e19458f5bbdae659ce77616dc0522e0eef706d9a5ef7b3ba2f7ed93a327c48\": container with ID starting with 87e19458f5bbdae659ce77616dc0522e0eef706d9a5ef7b3ba2f7ed93a327c48 not found: ID does not exist" containerID="87e19458f5bbdae659ce77616dc0522e0eef706d9a5ef7b3ba2f7ed93a327c48" Jul 24 04:10:51.056724 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:51.056684 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87e19458f5bbdae659ce77616dc0522e0eef706d9a5ef7b3ba2f7ed93a327c48"} err="failed to get container status \"87e19458f5bbdae659ce77616dc0522e0eef706d9a5ef7b3ba2f7ed93a327c48\": rpc error: code = NotFound desc = could not find container \"87e19458f5bbdae659ce77616dc0522e0eef706d9a5ef7b3ba2f7ed93a327c48\": container with ID starting with 87e19458f5bbdae659ce77616dc0522e0eef706d9a5ef7b3ba2f7ed93a327c48 not found: ID does not exist" Jul 24 04:10:51.060878 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:51.060856 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt"] Jul 24 04:10:51.065125 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:51.065104 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-5b56b9-openshift-default-6c664b86b4-czrnt"] Jul 24 04:10:51.316575 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:51.316435 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:10:53.033550 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:53.033515 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39ad2308-505c-4c87-9e7b-6a6508b52375" path="/var/lib/kubelet/pods/39ad2308-505c-4c87-9e7b-6a6508b52375/volumes" Jul 24 04:10:56.284218 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:56.284187 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:10:56.356771 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:56.356738 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:10:56.414523 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:10:56.414479 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:11:00.947717 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:00.947674 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:11:01.327972 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:01.327876 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:11:07.787183 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:07.787148 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57"] Jul 24 04:11:07.787684 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:07.787362 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" podUID="c4a8d26a-ac7f-4ea9-b909-4032697ad009" containerName="istio-proxy" containerID="cri-o://6ffe6e27f3323d4476ea6904237739ae99163c8731e79393f963129dbccdeb29" gracePeriod=30 Jul 24 04:11:08.343296 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:08.343246 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:11:08.461372 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:08.461334 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:11:13.043322 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.043298 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:11:13.138614 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.138587 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istio-podinfo\") pod \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " Jul 24 04:11:13.138614 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.138606 2579 generic.go:356] "Generic (PLEG): container finished" podID="c4a8d26a-ac7f-4ea9-b909-4032697ad009" containerID="6ffe6e27f3323d4476ea6904237739ae99163c8731e79393f963129dbccdeb29" exitCode=0 Jul 24 04:11:13.138839 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.138622 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-credential-socket\") pod \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " Jul 24 04:11:13.138839 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.138657 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6s8m\" (UniqueName: \"kubernetes.io/projected/c4a8d26a-ac7f-4ea9-b909-4032697ad009-kube-api-access-c6s8m\") pod \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " Jul 24 04:11:13.138839 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.138686 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istiod-ca-cert\") pod \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " Jul 24 04:11:13.138839 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.138706 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" event={"ID":"c4a8d26a-ac7f-4ea9-b909-4032697ad009","Type":"ContainerDied","Data":"6ffe6e27f3323d4476ea6904237739ae99163c8731e79393f963129dbccdeb29"} Jul 24 04:11:13.138839 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.138718 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" Jul 24 04:11:13.138839 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.138744 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57" event={"ID":"c4a8d26a-ac7f-4ea9-b909-4032697ad009","Type":"ContainerDied","Data":"5945bbef00b415a81007a8c84a4a44b98fbd07c2593ab332343774c6db006326"} Jul 24 04:11:13.138839 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.138765 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istio-token\") pod \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " Jul 24 04:11:13.138839 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.138773 2579 scope.go:117] "RemoveContainer" containerID="6ffe6e27f3323d4476ea6904237739ae99163c8731e79393f963129dbccdeb29" Jul 24 04:11:13.138839 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.138814 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-workload-socket\") pod \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " Jul 24 04:11:13.139330 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.138851 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istio-data\") pod \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " Jul 24 04:11:13.139330 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.138883 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-workload-certs\") pod \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " Jul 24 04:11:13.139330 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.138912 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istio-envoy\") pod \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\" (UID: \"c4a8d26a-ac7f-4ea9-b909-4032697ad009\") " Jul 24 04:11:13.139330 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.138939 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c4a8d26a-ac7f-4ea9-b909-4032697ad009" (UID: "c4a8d26a-ac7f-4ea9-b909-4032697ad009"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:11:13.139330 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.139071 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c4a8d26a-ac7f-4ea9-b909-4032697ad009" (UID: "c4a8d26a-ac7f-4ea9-b909-4032697ad009"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:11:13.139330 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.139230 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c4a8d26a-ac7f-4ea9-b909-4032697ad009" (UID: "c4a8d26a-ac7f-4ea9-b909-4032697ad009"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 04:11:13.139658 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.139352 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-workload-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:11:13.139658 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.139375 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-credential-socket\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:11:13.139658 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.139390 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istiod-ca-cert\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:11:13.139816 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.139725 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istio-data" (OuterVolumeSpecName: "istio-data") pod "c4a8d26a-ac7f-4ea9-b909-4032697ad009" (UID: "c4a8d26a-ac7f-4ea9-b909-4032697ad009"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:11:13.139960 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.139934 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c4a8d26a-ac7f-4ea9-b909-4032697ad009" (UID: "c4a8d26a-ac7f-4ea9-b909-4032697ad009"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:11:13.141303 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.141277 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a8d26a-ac7f-4ea9-b909-4032697ad009-kube-api-access-c6s8m" (OuterVolumeSpecName: "kube-api-access-c6s8m") pod "c4a8d26a-ac7f-4ea9-b909-4032697ad009" (UID: "c4a8d26a-ac7f-4ea9-b909-4032697ad009"). InnerVolumeSpecName "kube-api-access-c6s8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 04:11:13.141520 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.141496 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c4a8d26a-ac7f-4ea9-b909-4032697ad009" (UID: "c4a8d26a-ac7f-4ea9-b909-4032697ad009"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 04:11:13.142014 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.141987 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c4a8d26a-ac7f-4ea9-b909-4032697ad009" (UID: "c4a8d26a-ac7f-4ea9-b909-4032697ad009"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 04:11:13.142342 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.142315 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istio-token" (OuterVolumeSpecName: "istio-token") pod "c4a8d26a-ac7f-4ea9-b909-4032697ad009" (UID: "c4a8d26a-ac7f-4ea9-b909-4032697ad009"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 04:11:13.160753 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.160735 2579 scope.go:117] "RemoveContainer" containerID="a1f5d773f7832753012c357ec03b52d521987487fbc2f8ac56dd2b6f69dc8e4f" Jul 24 04:11:13.168776 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.168760 2579 scope.go:117] "RemoveContainer" containerID="6ffe6e27f3323d4476ea6904237739ae99163c8731e79393f963129dbccdeb29" Jul 24 04:11:13.169019 ip-10-0-134-147 kubenswrapper[2579]: E0724 04:11:13.168999 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ffe6e27f3323d4476ea6904237739ae99163c8731e79393f963129dbccdeb29\": container with ID starting with 6ffe6e27f3323d4476ea6904237739ae99163c8731e79393f963129dbccdeb29 not found: ID does not exist" containerID="6ffe6e27f3323d4476ea6904237739ae99163c8731e79393f963129dbccdeb29" Jul 24 04:11:13.169087 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.169031 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ffe6e27f3323d4476ea6904237739ae99163c8731e79393f963129dbccdeb29"} err="failed to get container status \"6ffe6e27f3323d4476ea6904237739ae99163c8731e79393f963129dbccdeb29\": rpc error: code = NotFound desc = could not find container \"6ffe6e27f3323d4476ea6904237739ae99163c8731e79393f963129dbccdeb29\": container with ID starting with 6ffe6e27f3323d4476ea6904237739ae99163c8731e79393f963129dbccdeb29 not found: ID does not exist" Jul 24 04:11:13.169087 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.169054 2579 scope.go:117] "RemoveContainer" containerID="a1f5d773f7832753012c357ec03b52d521987487fbc2f8ac56dd2b6f69dc8e4f" Jul 24 04:11:13.169281 ip-10-0-134-147 kubenswrapper[2579]: E0724 04:11:13.169261 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1f5d773f7832753012c357ec03b52d521987487fbc2f8ac56dd2b6f69dc8e4f\": container with ID starting with a1f5d773f7832753012c357ec03b52d521987487fbc2f8ac56dd2b6f69dc8e4f not found: ID does not exist" containerID="a1f5d773f7832753012c357ec03b52d521987487fbc2f8ac56dd2b6f69dc8e4f" Jul 24 04:11:13.169338 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.169290 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f5d773f7832753012c357ec03b52d521987487fbc2f8ac56dd2b6f69dc8e4f"} err="failed to get container status \"a1f5d773f7832753012c357ec03b52d521987487fbc2f8ac56dd2b6f69dc8e4f\": rpc error: code = NotFound desc = could not find container \"a1f5d773f7832753012c357ec03b52d521987487fbc2f8ac56dd2b6f69dc8e4f\": container with ID starting with a1f5d773f7832753012c357ec03b52d521987487fbc2f8ac56dd2b6f69dc8e4f not found: ID does not exist" Jul 24 04:11:13.239907 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.239876 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istio-token\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:11:13.239907 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.239903 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istio-data\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:11:13.240072 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.239915 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-workload-certs\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:11:13.240072 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.239926 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istio-envoy\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:11:13.240072 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.239937 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4a8d26a-ac7f-4ea9-b909-4032697ad009-istio-podinfo\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:11:13.240072 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.239948 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6s8m\" (UniqueName: \"kubernetes.io/projected/c4a8d26a-ac7f-4ea9-b909-4032697ad009-kube-api-access-c6s8m\") on node \"ip-10-0-134-147.ec2.internal\" DevicePath \"\"" Jul 24 04:11:13.461361 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.461331 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57"] Jul 24 04:11:13.465325 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:13.465299 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-f701e0-openshift-default-58679d885b-b6z57"] Jul 24 04:11:14.198187 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:14.198154 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-dsqpq_06e7bf12-474c-4a51-be97-44cc726803f2/manager/0.log" Jul 24 04:11:14.322380 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:14.322352 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-zhwkc_2b03c41a-0004-44ea-ab51-ac94dde9c9a0/manager/0.log" Jul 24 04:11:14.448041 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:14.448006 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-687994cdfd-m8bvr_c4998e24-36bc-49d8-968d-0f60ae219e0b/manager/0.log" Jul 24 04:11:14.574435 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:14.574364 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-cngtc_56d05014-f053-447d-b1d2-37b530a3ef52/server/0.log" Jul 24 04:11:14.700005 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:14.699977 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9cn76_aa9efd80-0fe0-4b2a-aff1-1a516a4dcc2b/manager/0.log" Jul 24 04:11:14.822952 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:14.822921 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67f4587fcc-44l4m_56b90bb3-b363-4be6-ba7c-6c5c10ff8b06/copy-manifests/0.log" Jul 24 04:11:14.835561 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:14.835491 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67f4587fcc-44l4m_56b90bb3-b363-4be6-ba7c-6c5c10ff8b06/manager/0.log" Jul 24 04:11:15.035225 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:15.035182 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4a8d26a-ac7f-4ea9-b909-4032697ad009" path="/var/lib/kubelet/pods/c4a8d26a-ac7f-4ea9-b909-4032697ad009/volumes" Jul 24 04:11:15.484625 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:15.484593 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-857c5484bd-xlxc2_810d712e-8532-4bc5-a2a4-12b2b2f37330/maas-api/0.log" Jul 24 04:11:15.610755 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:15.610723 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29747745-xzbmt_1610a97e-cf52-4955-a551-8de63aedf990/cleanup/0.log" Jul 24 04:11:15.860052 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:15.860024 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-4f8cw_86e1bda7-4a0a-428f-8cf8-76ba06dc7fb9/postgres/0.log" Jul 24 04:11:17.359346 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:17.359304 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-kksmh_5848e713-d744-4ecd-baf7-40d3bdf70db6/manager/0.log" Jul 24 04:11:17.481139 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:17.481110 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-twsqn_7c5ca5e8-c8b2-4f7c-a7e8-43594cc7635a/manager/0.log" Jul 24 04:11:17.601798 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:17.601774 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-xq4p7_dbc8994c-8f77-41e7-ada1-8aaa9a85de74/kuadrant-console-plugin/0.log" Jul 24 04:11:17.725861 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:17.725782 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-ttwx2_3e799fcf-1581-48ce-94c2-f8c977bf6d42/registry-server/0.log" Jul 24 04:11:18.105995 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:18.105917 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-dmrst_fc166356-d039-4390-9c9e-8b117b744b3f/manager/0.log" Jul 24 04:11:18.487957 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:18.487921 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp_6265fa7d-16b0-46c6-868c-2200a454e4b3/istio-proxy/0.log" Jul 24 04:11:18.849827 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:18.849741 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f459d4668-c22g9_b5b6f8d8-a71c-4088-bc54-52634f3d8eee/kube-auth-proxy/0.log" Jul 24 04:11:18.977138 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:18.977110 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-8s6w4_e6304eb4-8cda-4f2b-8d31-b2da9c27a10a/istio-proxy/1.log" Jul 24 04:11:19.100884 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:19.100806 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-fczls_5e37496f-fe2b-4acb-97df-faefd815ca0d/payload-pre-processing/0.log" Jul 24 04:11:19.702302 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:19.702272 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-l8mtj_30343da6-f4a4-4bdc-be6c-f4978d409439/main/0.log" Jul 24 04:11:19.934327 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:19.934298 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-cvztf_2078c0a8-340d-471d-8d2c-a022813c0ef9/main/0.log" Jul 24 04:11:20.284155 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:20.284127 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-bw6vp_103d6fe6-8801-44bd-9b7a-f1a9ad93080d/main/0.log" Jul 24 04:11:25.708673 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:25.708633 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:11:25.807554 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:25.807526 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:11:28.548401 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:28.548369 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fhm9h_ad666805-ba79-4b67-8a3b-b9ba32afe705/global-pull-secret-syncer/0.log" Jul 24 04:11:28.596975 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:28.596951 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" Jul 24 04:11:28.712001 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:28.711975 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-147.ec2.internal_e7160710802e7696127f93542dcf0579/haproxy/0.log" Jul 24 04:11:34.271730 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:34.271704 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-kksmh_5848e713-d744-4ecd-baf7-40d3bdf70db6/manager/0.log" Jul 24 04:11:34.312767 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:34.312728 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-twsqn_7c5ca5e8-c8b2-4f7c-a7e8-43594cc7635a/manager/0.log" Jul 24 04:11:34.338381 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:34.338356 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-xq4p7_dbc8994c-8f77-41e7-ada1-8aaa9a85de74/kuadrant-console-plugin/0.log" Jul 24 04:11:34.389589 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:34.389566 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-ttwx2_3e799fcf-1581-48ce-94c2-f8c977bf6d42/registry-server/0.log" Jul 24 04:11:34.632295 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:34.632261 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-dmrst_fc166356-d039-4390-9c9e-8b117b744b3f/manager/0.log" Jul 24 04:11:36.735775 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:36.735746 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-82q8v_31e8d580-562b-482b-9c7b-673cf95070ea/node-exporter/0.log" Jul 24 04:11:36.757670 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:36.757645 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-82q8v_31e8d580-562b-482b-9c7b-673cf95070ea/kube-rbac-proxy/0.log" Jul 24 04:11:36.780259 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:36.780235 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-82q8v_31e8d580-562b-482b-9c7b-673cf95070ea/init-textfile/0.log" Jul 24 04:11:37.218712 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:37.218684 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-lh52r_ab8bad1f-dacb-4db0-af0b-3efb426a5405/prometheus-operator-admission-webhook/0.log" Jul 24 04:11:37.249681 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:37.249651 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-58469cd6d-xrpbr_d1c880c3-5228-4b9d-a829-04cb5c44f37b/telemeter-client/0.log" Jul 24 04:11:37.272112 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:37.272085 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-58469cd6d-xrpbr_d1c880c3-5228-4b9d-a829-04cb5c44f37b/reload/0.log" Jul 24 04:11:37.294688 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:37.294661 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-58469cd6d-xrpbr_d1c880c3-5228-4b9d-a829-04cb5c44f37b/kube-rbac-proxy/0.log" Jul 24 04:11:38.670955 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.670921 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tsxt7/perf-node-gather-daemonset-qvqnp"] Jul 24 04:11:38.671390 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.671346 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4a8d26a-ac7f-4ea9-b909-4032697ad009" containerName="istio-proxy" Jul 24 04:11:38.671390 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.671357 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4a8d26a-ac7f-4ea9-b909-4032697ad009" containerName="istio-proxy" Jul 24 04:11:38.671390 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.671366 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39ad2308-505c-4c87-9e7b-6a6508b52375" containerName="istio-proxy" Jul 24 04:11:38.671390 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.671372 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="39ad2308-505c-4c87-9e7b-6a6508b52375" containerName="istio-proxy" Jul 24 04:11:38.671390 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.671378 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57" containerName="istio-proxy" Jul 24 04:11:38.671390 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.671383 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57" containerName="istio-proxy" Jul 24 04:11:38.671390 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.671393 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc51c7c5-0c40-4f65-9da6-213029264e3b" containerName="istio-proxy" Jul 24 04:11:38.671637 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.671400 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc51c7c5-0c40-4f65-9da6-213029264e3b" containerName="istio-proxy" Jul 24 04:11:38.671637 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.671408 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cc0aea7-aecb-4e5f-9453-61feb2205530" containerName="istio-proxy" Jul 24 04:11:38.671637 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.671413 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cc0aea7-aecb-4e5f-9453-61feb2205530" containerName="istio-proxy" Jul 24 04:11:38.671637 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.671420 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc51c7c5-0c40-4f65-9da6-213029264e3b" containerName="istio-proxy" Jul 24 04:11:38.671637 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.671426 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc51c7c5-0c40-4f65-9da6-213029264e3b" containerName="istio-proxy" Jul 24 04:11:38.671637 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.671438 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57" containerName="istio-proxy" Jul 24 04:11:38.671637 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.671442 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57" containerName="istio-proxy" Jul 24 04:11:38.671637 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.671526 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc51c7c5-0c40-4f65-9da6-213029264e3b" containerName="istio-proxy" Jul 24 04:11:38.671637 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.671534 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc51c7c5-0c40-4f65-9da6-213029264e3b" containerName="istio-proxy" Jul 24 04:11:38.671637 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.671542 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="07a38ba5-6d7b-4dd9-87ec-1992ba1f2c57" containerName="istio-proxy" Jul 24 04:11:38.671637 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.671547 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cc0aea7-aecb-4e5f-9453-61feb2205530" containerName="istio-proxy" Jul 24 04:11:38.671637 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.671553 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="39ad2308-505c-4c87-9e7b-6a6508b52375" containerName="istio-proxy" Jul 24 04:11:38.671637 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.671560 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4a8d26a-ac7f-4ea9-b909-4032697ad009" containerName="istio-proxy" Jul 24 04:11:38.671637 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.671566 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cc0aea7-aecb-4e5f-9453-61feb2205530" containerName="istio-proxy" Jul 24 04:11:38.671637 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.671573 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4a8d26a-ac7f-4ea9-b909-4032697ad009" containerName="istio-proxy" Jul 24 04:11:38.675998 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.675976 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tsxt7/perf-node-gather-daemonset-qvqnp" Jul 24 04:11:38.678057 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.678039 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tsxt7\"/\"kube-root-ca.crt\"" Jul 24 04:11:38.678165 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.678046 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tsxt7\"/\"default-dockercfg-jw6nv\"" Jul 24 04:11:38.678962 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.678933 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tsxt7\"/\"openshift-service-ca.crt\"" Jul 24 04:11:38.684100 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.684077 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tsxt7/perf-node-gather-daemonset-qvqnp"] Jul 24 04:11:38.781745 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.781712 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e15ea9ff-804a-4c81-94f0-d2e910bb6a91-sys\") pod \"perf-node-gather-daemonset-qvqnp\" (UID: \"e15ea9ff-804a-4c81-94f0-d2e910bb6a91\") " pod="openshift-must-gather-tsxt7/perf-node-gather-daemonset-qvqnp" Jul 24 04:11:38.781907 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.781761 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb9ss\" (UniqueName: \"kubernetes.io/projected/e15ea9ff-804a-4c81-94f0-d2e910bb6a91-kube-api-access-xb9ss\") pod \"perf-node-gather-daemonset-qvqnp\" (UID: \"e15ea9ff-804a-4c81-94f0-d2e910bb6a91\") " pod="openshift-must-gather-tsxt7/perf-node-gather-daemonset-qvqnp" Jul 24 04:11:38.781907 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.781840 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e15ea9ff-804a-4c81-94f0-d2e910bb6a91-lib-modules\") pod \"perf-node-gather-daemonset-qvqnp\" (UID: \"e15ea9ff-804a-4c81-94f0-d2e910bb6a91\") " pod="openshift-must-gather-tsxt7/perf-node-gather-daemonset-qvqnp" Jul 24 04:11:38.781987 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.781916 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e15ea9ff-804a-4c81-94f0-d2e910bb6a91-podres\") pod \"perf-node-gather-daemonset-qvqnp\" (UID: \"e15ea9ff-804a-4c81-94f0-d2e910bb6a91\") " pod="openshift-must-gather-tsxt7/perf-node-gather-daemonset-qvqnp" Jul 24 04:11:38.781987 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.781958 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e15ea9ff-804a-4c81-94f0-d2e910bb6a91-proc\") pod \"perf-node-gather-daemonset-qvqnp\" (UID: \"e15ea9ff-804a-4c81-94f0-d2e910bb6a91\") " pod="openshift-must-gather-tsxt7/perf-node-gather-daemonset-qvqnp" Jul 24 04:11:38.882895 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.882860 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e15ea9ff-804a-4c81-94f0-d2e910bb6a91-proc\") pod \"perf-node-gather-daemonset-qvqnp\" (UID: \"e15ea9ff-804a-4c81-94f0-d2e910bb6a91\") " pod="openshift-must-gather-tsxt7/perf-node-gather-daemonset-qvqnp" Jul 24 04:11:38.883058 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.882905 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e15ea9ff-804a-4c81-94f0-d2e910bb6a91-sys\") pod \"perf-node-gather-daemonset-qvqnp\" (UID: \"e15ea9ff-804a-4c81-94f0-d2e910bb6a91\") " pod="openshift-must-gather-tsxt7/perf-node-gather-daemonset-qvqnp" Jul 24 04:11:38.883058 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.882981 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e15ea9ff-804a-4c81-94f0-d2e910bb6a91-sys\") pod \"perf-node-gather-daemonset-qvqnp\" (UID: \"e15ea9ff-804a-4c81-94f0-d2e910bb6a91\") " pod="openshift-must-gather-tsxt7/perf-node-gather-daemonset-qvqnp" Jul 24 04:11:38.883058 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.882985 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e15ea9ff-804a-4c81-94f0-d2e910bb6a91-proc\") pod \"perf-node-gather-daemonset-qvqnp\" (UID: \"e15ea9ff-804a-4c81-94f0-d2e910bb6a91\") " pod="openshift-must-gather-tsxt7/perf-node-gather-daemonset-qvqnp" Jul 24 04:11:38.883058 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.883006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb9ss\" (UniqueName: \"kubernetes.io/projected/e15ea9ff-804a-4c81-94f0-d2e910bb6a91-kube-api-access-xb9ss\") pod \"perf-node-gather-daemonset-qvqnp\" (UID: \"e15ea9ff-804a-4c81-94f0-d2e910bb6a91\") " pod="openshift-must-gather-tsxt7/perf-node-gather-daemonset-qvqnp" Jul 24 04:11:38.883058 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.883034 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e15ea9ff-804a-4c81-94f0-d2e910bb6a91-lib-modules\") pod \"perf-node-gather-daemonset-qvqnp\" (UID: \"e15ea9ff-804a-4c81-94f0-d2e910bb6a91\") " pod="openshift-must-gather-tsxt7/perf-node-gather-daemonset-qvqnp" Jul 24 04:11:38.883339 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.883150 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e15ea9ff-804a-4c81-94f0-d2e910bb6a91-lib-modules\") pod \"perf-node-gather-daemonset-qvqnp\" (UID: \"e15ea9ff-804a-4c81-94f0-d2e910bb6a91\") " pod="openshift-must-gather-tsxt7/perf-node-gather-daemonset-qvqnp" Jul 24 04:11:38.883339 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.883203 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e15ea9ff-804a-4c81-94f0-d2e910bb6a91-podres\") pod \"perf-node-gather-daemonset-qvqnp\" (UID: \"e15ea9ff-804a-4c81-94f0-d2e910bb6a91\") " pod="openshift-must-gather-tsxt7/perf-node-gather-daemonset-qvqnp" Jul 24 04:11:38.883339 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.883309 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e15ea9ff-804a-4c81-94f0-d2e910bb6a91-podres\") pod \"perf-node-gather-daemonset-qvqnp\" (UID: \"e15ea9ff-804a-4c81-94f0-d2e910bb6a91\") " pod="openshift-must-gather-tsxt7/perf-node-gather-daemonset-qvqnp" Jul 24 04:11:38.891398 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.891361 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb9ss\" (UniqueName: \"kubernetes.io/projected/e15ea9ff-804a-4c81-94f0-d2e910bb6a91-kube-api-access-xb9ss\") pod \"perf-node-gather-daemonset-qvqnp\" (UID: \"e15ea9ff-804a-4c81-94f0-d2e910bb6a91\") " pod="openshift-must-gather-tsxt7/perf-node-gather-daemonset-qvqnp" Jul 24 04:11:38.986996 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:38.986921 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tsxt7/perf-node-gather-daemonset-qvqnp" Jul 24 04:11:39.118770 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:39.118747 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tsxt7/perf-node-gather-daemonset-qvqnp"] Jul 24 04:11:39.120681 ip-10-0-134-147 kubenswrapper[2579]: W0724 04:11:39.120654 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode15ea9ff_804a_4c81_94f0_d2e910bb6a91.slice/crio-bbaa87c61b123185479c47b76ca8e555d28fa18cc11198bc3a083b5c7f9cee38 WatchSource:0}: Error finding container bbaa87c61b123185479c47b76ca8e555d28fa18cc11198bc3a083b5c7f9cee38: Status 404 returned error can't find the container with id bbaa87c61b123185479c47b76ca8e555d28fa18cc11198bc3a083b5c7f9cee38 Jul 24 04:11:39.275143 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:39.275046 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tsxt7/perf-node-gather-daemonset-qvqnp" event={"ID":"e15ea9ff-804a-4c81-94f0-d2e910bb6a91","Type":"ContainerStarted","Data":"06737b5c2f8e7bf477da80a93aa93048dffc6d02105d7018ed6a92c4384cbd49"} Jul 24 04:11:39.275143 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:39.275103 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tsxt7/perf-node-gather-daemonset-qvqnp" event={"ID":"e15ea9ff-804a-4c81-94f0-d2e910bb6a91","Type":"ContainerStarted","Data":"bbaa87c61b123185479c47b76ca8e555d28fa18cc11198bc3a083b5c7f9cee38"} Jul 24 04:11:39.275143 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:39.275129 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tsxt7/perf-node-gather-daemonset-qvqnp" Jul 24 04:11:39.291421 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:39.291372 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tsxt7/perf-node-gather-daemonset-qvqnp" podStartSLOduration=1.291356843 podStartE2EDuration="1.291356843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 04:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 04:11:39.289957412 +0000 UTC m=+3248.729376875" watchObservedRunningTime="2026-07-24 04:11:39.291356843 +0000 UTC m=+3248.730776325" Jul 24 04:11:41.417376 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:41.417340 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4zq5x_1a36f9a1-1cc8-4858-be3f-ec11e34eebab/dns/0.log" Jul 24 04:11:41.452485 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:41.452441 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4zq5x_1a36f9a1-1cc8-4858-be3f-ec11e34eebab/kube-rbac-proxy/0.log" Jul 24 04:11:41.653818 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:41.653790 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zgsks_a5b9b325-d029-42ca-8dfb-c75c2dd0c726/dns-node-resolver/0.log" Jul 24 04:11:42.055922 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:42.055887 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9g9rf_ce6090ae-6c38-4fb6-b7dd-9d76baa4951b/konnectivity-agent/0.log" limit=1024 Jul 24 04:11:42.190181 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:42.190153 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-77678456c6-kmzrp_003896e6-6475-4611-92c7-9beea3b43ae9/registry/0.log" Jul 24 04:11:42.265887 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:42.265862 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8brgv_daacf9bc-9977-408a-b1b4-cd361f3cf528/node-ca/0.log" Jul 24 04:11:43.260398 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:43.260371 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp_6265fa7d-16b0-46c6-868c-2200a454e4b3/istio-proxy/0.log" Jul 24 04:11:43.453815 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:43.453783 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f459d4668-c22g9_b5b6f8d8-a71c-4088-bc54-52634f3d8eee/kube-auth-proxy/0.log" Jul 24 04:11:43.485173 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:43.485149 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-8s6w4_e6304eb4-8cda-4f2b-8d31-b2da9c27a10a/istio-proxy/0.log" Jul 24 04:11:43.510082 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:43.510056 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-8s6w4_e6304eb4-8cda-4f2b-8d31-b2da9c27a10a/istio-proxy/1.log" Jul 24 04:11:43.540446 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:43.540380 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-fczls_5e37496f-fe2b-4acb-97df-faefd815ca0d/payload-pre-processing/0.log" Jul 24 04:11:44.476656 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:44.476627 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-75ljs_e63f0954-7da9-4fdc-a43d-d1cc3323aaa1/serve-healthcheck-canary/0.log" Jul 24 04:11:44.979580 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:44.979524 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-8gpjf_587fc069-a2aa-4701-82e9-65e01ab74ade/insights-operator/0.log" Jul 24 04:11:44.980388 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:44.980369 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-8gpjf_587fc069-a2aa-4701-82e9-65e01ab74ade/insights-operator/1.log" Jul 24 04:11:45.163024 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:45.162996 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xnpz8_9373ab5f-eb4b-495c-9f7f-b0f00e884339/kube-rbac-proxy/0.log" Jul 24 04:11:45.185493 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:45.185452 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xnpz8_9373ab5f-eb4b-495c-9f7f-b0f00e884339/exporter/0.log" Jul 24 04:11:45.209291 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:45.209270 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xnpz8_9373ab5f-eb4b-495c-9f7f-b0f00e884339/extractor/0.log" Jul 24 04:11:45.290657 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:45.290603 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tsxt7/perf-node-gather-daemonset-qvqnp" Jul 24 04:11:47.412507 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:47.412478 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-dsqpq_06e7bf12-474c-4a51-be97-44cc726803f2/manager/0.log" Jul 24 04:11:47.607088 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:47.607005 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-zhwkc_2b03c41a-0004-44ea-ab51-ac94dde9c9a0/manager/0.log" Jul 24 04:11:47.735384 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:47.735356 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-687994cdfd-m8bvr_c4998e24-36bc-49d8-968d-0f60ae219e0b/manager/0.log" Jul 24 04:11:47.755318 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:47.755297 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-cngtc_56d05014-f053-447d-b1d2-37b530a3ef52/server/0.log" Jul 24 04:11:47.831247 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:47.831215 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9cn76_aa9efd80-0fe0-4b2a-aff1-1a516a4dcc2b/manager/0.log" Jul 24 04:11:47.912813 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:47.912730 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67f4587fcc-44l4m_56b90bb3-b363-4be6-ba7c-6c5c10ff8b06/manager/0.log" Jul 24 04:11:47.937916 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:47.937896 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67f4587fcc-44l4m_56b90bb3-b363-4be6-ba7c-6c5c10ff8b06/copy-manifests/0.log" Jul 24 04:11:49.436035 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:49.436003 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-27zv2_8187f663-7660-481e-a5d7-4f5b1d6a5179/openshift-lws-operator/0.log" Jul 24 04:11:49.566375 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:49.566347 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp_6265fa7d-16b0-46c6-868c-2200a454e4b3/istio-proxy/0.log" Jul 24 04:11:49.729014 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:49.728931 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f459d4668-c22g9_b5b6f8d8-a71c-4088-bc54-52634f3d8eee/kube-auth-proxy/0.log" Jul 24 04:11:49.753182 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:49.753158 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-8s6w4_e6304eb4-8cda-4f2b-8d31-b2da9c27a10a/istio-proxy/0.log" Jul 24 04:11:49.780216 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:49.780186 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-8s6w4_e6304eb4-8cda-4f2b-8d31-b2da9c27a10a/istio-proxy/1.log" Jul 24 04:11:49.800837 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:49.800810 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-fczls_5e37496f-fe2b-4acb-97df-faefd815ca0d/payload-pre-processing/0.log" Jul 24 04:11:51.260952 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:51.260923 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c8pdcp_6265fa7d-16b0-46c6-868c-2200a454e4b3/istio-proxy/0.log" Jul 24 04:11:51.446146 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:51.446117 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f459d4668-c22g9_b5b6f8d8-a71c-4088-bc54-52634f3d8eee/kube-auth-proxy/0.log" Jul 24 04:11:51.466542 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:51.466512 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-8s6w4_e6304eb4-8cda-4f2b-8d31-b2da9c27a10a/istio-proxy/0.log" Jul 24 04:11:51.499420 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:51.499387 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-8s6w4_e6304eb4-8cda-4f2b-8d31-b2da9c27a10a/istio-proxy/1.log" Jul 24 04:11:51.524126 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:51.524011 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-fczls_5e37496f-fe2b-4acb-97df-faefd815ca0d/payload-pre-processing/0.log" Jul 24 04:11:54.615438 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:54.615403 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-tsp28_12956bc1-3664-46a8-97e6-be81968535d9/migrator/0.log" Jul 24 04:11:54.637787 ip-10-0-134-147 kubenswrapper[2579]: I0724 04:11:54.637759 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-tsp28_12956bc1-3664-46a8-97e6-be81968535d9/graceful-termination/0.log"