Jul 30 00:01:52.383416 ip-10-0-135-118 systemd[1]: Starting Kubernetes Kubelet... Jul 30 00:01:52.771408 ip-10-0-135-118 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 00:01:52.771408 ip-10-0-135-118 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 30 00:01:52.771408 ip-10-0-135-118 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 00:01:52.771408 ip-10-0-135-118 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 30 00:01:52.771408 ip-10-0-135-118 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 00:01:52.772332 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.772244 2572 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 30 00:01:52.776389 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776374 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 00:01:52.776428 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776390 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 00:01:52.776428 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776394 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 00:01:52.776428 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776399 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 00:01:52.776428 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776402 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 00:01:52.776428 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776405 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 00:01:52.776428 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776408 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 00:01:52.776428 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776411 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 00:01:52.776428 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776414 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 00:01:52.776428 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776416 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 00:01:52.776428 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776419 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 00:01:52.776428 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776423 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 00:01:52.776428 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776425 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 00:01:52.776428 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776428 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 00:01:52.776428 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776431 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 00:01:52.776428 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776434 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 00:01:52.776795 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776437 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 00:01:52.776795 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776440 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 00:01:52.776795 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776443 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 30 00:01:52.776795 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776445 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 00:01:52.776795 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776448 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 00:01:52.776795 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776451 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 00:01:52.776795 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776455 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 00:01:52.776795 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776459 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 00:01:52.776795 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776462 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 00:01:52.776795 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776465 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 00:01:52.776795 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776469 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 00:01:52.776795 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776471 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 00:01:52.776795 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776474 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 00:01:52.776795 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776476 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 00:01:52.776795 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776479 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 00:01:52.776795 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776482 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 00:01:52.776795 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776485 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 00:01:52.776795 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776489 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 00:01:52.776795 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776492 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 00:01:52.777275 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776496 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 00:01:52.777275 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776499 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 00:01:52.777275 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776502 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 00:01:52.777275 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776504 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 00:01:52.777275 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776507 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 00:01:52.777275 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776509 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 00:01:52.777275 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776512 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 00:01:52.777275 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776515 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 00:01:52.777275 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776518 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 00:01:52.777275 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776520 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 00:01:52.777275 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776523 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 00:01:52.777275 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776525 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 00:01:52.777275 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776528 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 00:01:52.777275 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776530 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 00:01:52.777275 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776534 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 00:01:52.777275 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776536 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 00:01:52.777275 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776539 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 00:01:52.777275 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776542 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 00:01:52.777275 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776545 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 00:01:52.777275 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776548 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 00:01:52.777771 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776550 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 00:01:52.777771 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776553 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 00:01:52.777771 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776556 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 00:01:52.777771 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776558 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 00:01:52.777771 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776561 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 00:01:52.777771 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776563 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 00:01:52.777771 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776566 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 00:01:52.777771 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776569 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 00:01:52.777771 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776571 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 00:01:52.777771 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776574 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 00:01:52.777771 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776576 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 00:01:52.777771 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776579 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 00:01:52.777771 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776582 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 00:01:52.777771 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776584 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 00:01:52.777771 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776587 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 00:01:52.777771 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776589 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 00:01:52.777771 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776592 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 00:01:52.777771 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776595 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 00:01:52.777771 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776598 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 00:01:52.778294 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776601 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 00:01:52.778294 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776603 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 00:01:52.778294 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776605 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 00:01:52.778294 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776608 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 00:01:52.778294 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776611 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 00:01:52.778294 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776613 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 00:01:52.778294 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776616 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 00:01:52.778294 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776621 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 00:01:52.778294 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776625 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 00:01:52.778294 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776628 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 00:01:52.778294 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776631 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 00:01:52.778294 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776634 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 00:01:52.778294 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776636 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 00:01:52.778294 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776639 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 00:01:52.778294 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776642 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 00:01:52.778294 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776644 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 00:01:52.778294 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776647 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 00:01:52.778294 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776650 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 00:01:52.778294 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776652 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 00:01:52.778756 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776654 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 00:01:52.778756 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776657 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 00:01:52.778756 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776659 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 00:01:52.778756 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776662 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 00:01:52.778756 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776664 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 00:01:52.778756 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.776667 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 00:01:52.778756 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777381 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 00:01:52.778756 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777388 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 00:01:52.778756 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777391 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 00:01:52.778756 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777394 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 00:01:52.778756 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777396 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 00:01:52.778756 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777400 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 00:01:52.778756 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777403 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 00:01:52.778756 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777405 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 00:01:52.778756 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777408 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 00:01:52.778756 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777410 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 00:01:52.778756 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777413 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 00:01:52.778756 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777416 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 00:01:52.778756 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777418 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 00:01:52.778756 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777421 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 00:01:52.779271 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777425 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 00:01:52.779271 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777427 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 00:01:52.779271 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777430 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 00:01:52.779271 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777433 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 00:01:52.779271 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777435 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 00:01:52.779271 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777440 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 00:01:52.779271 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777443 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 00:01:52.779271 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777446 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 00:01:52.779271 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777449 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 00:01:52.779271 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777452 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 00:01:52.779271 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777455 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 00:01:52.779271 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777458 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 00:01:52.779271 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777460 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 00:01:52.779271 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777463 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 00:01:52.779271 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777466 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 00:01:52.779271 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777468 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 00:01:52.779271 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777471 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 00:01:52.779271 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777474 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 00:01:52.779271 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777477 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 00:01:52.779752 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777481 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 00:01:52.779752 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777484 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 00:01:52.779752 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777486 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 00:01:52.779752 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777489 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 00:01:52.779752 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777491 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 00:01:52.779752 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777494 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 00:01:52.779752 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777497 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 00:01:52.779752 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777500 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 00:01:52.779752 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777502 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 00:01:52.779752 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777505 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 00:01:52.779752 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777508 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 00:01:52.779752 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777510 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 00:01:52.779752 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777513 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 00:01:52.779752 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777517 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 30 00:01:52.779752 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777519 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 00:01:52.779752 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777522 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 00:01:52.779752 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777524 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 00:01:52.779752 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777527 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 00:01:52.779752 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777530 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 00:01:52.779752 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777533 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 00:01:52.780326 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777536 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 00:01:52.780326 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777538 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 00:01:52.780326 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777541 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 00:01:52.780326 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777545 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 00:01:52.780326 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777548 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 00:01:52.780326 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777550 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 00:01:52.780326 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777553 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 00:01:52.780326 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777555 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 00:01:52.780326 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777558 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 00:01:52.780326 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777561 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 00:01:52.780326 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777563 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 00:01:52.780326 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777566 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 00:01:52.780326 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777568 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 00:01:52.780326 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777571 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 00:01:52.780326 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777574 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 00:01:52.780326 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777576 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 00:01:52.780326 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777579 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 00:01:52.780326 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777582 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 00:01:52.780326 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777587 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 00:01:52.780810 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777590 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 00:01:52.780810 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777593 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 00:01:52.780810 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777596 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 00:01:52.780810 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777599 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 00:01:52.780810 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777602 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 00:01:52.780810 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777604 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 00:01:52.780810 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777608 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 00:01:52.780810 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777611 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 00:01:52.780810 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777614 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 00:01:52.780810 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777616 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 00:01:52.780810 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777619 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 00:01:52.780810 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777621 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 00:01:52.780810 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777624 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 00:01:52.780810 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777627 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 00:01:52.780810 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777629 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 00:01:52.780810 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777631 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 00:01:52.780810 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777634 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 00:01:52.780810 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777637 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 00:01:52.780810 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777639 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 00:01:52.780810 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777641 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 00:01:52.781332 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777644 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 00:01:52.781332 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777646 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 00:01:52.781332 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777649 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 00:01:52.781332 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777651 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 00:01:52.781332 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777654 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 00:01:52.781332 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777656 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 00:01:52.781332 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.777659 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 00:01:52.781332 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778076 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 00:01:52.781332 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778083 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 30 00:01:52.781332 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778086 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 00:01:52.781332 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778089 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 00:01:52.781332 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778092 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 00:01:52.781332 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778095 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 00:01:52.781332 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778098 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 00:01:52.781332 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778100 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 00:01:52.781332 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778103 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 00:01:52.781332 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778120 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 00:01:52.781332 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778123 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 00:01:52.781332 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778126 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 00:01:52.781332 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778129 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 00:01:52.781823 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778132 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 00:01:52.781823 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778135 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 00:01:52.781823 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778138 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 00:01:52.781823 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778140 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 00:01:52.781823 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778143 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 00:01:52.781823 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778146 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 00:01:52.781823 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778149 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 00:01:52.781823 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778152 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 00:01:52.781823 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778156 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 00:01:52.781823 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778160 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 00:01:52.781823 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778163 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 00:01:52.781823 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778166 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 00:01:52.781823 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778169 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 00:01:52.781823 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778171 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 00:01:52.781823 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778174 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 00:01:52.781823 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778177 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 00:01:52.781823 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778179 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 00:01:52.781823 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778182 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 00:01:52.781823 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778184 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 00:01:52.782295 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778187 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 00:01:52.782295 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778189 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 00:01:52.782295 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778192 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 00:01:52.782295 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778194 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 00:01:52.782295 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778197 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 00:01:52.782295 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778200 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 00:01:52.782295 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778203 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 00:01:52.782295 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778205 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 00:01:52.782295 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778208 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 00:01:52.782295 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778211 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 00:01:52.782295 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778215 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 00:01:52.782295 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778220 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 00:01:52.782295 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778224 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 00:01:52.782295 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778228 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 00:01:52.782295 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778231 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 00:01:52.782295 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778234 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 00:01:52.782295 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778237 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 00:01:52.782295 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778239 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 00:01:52.782295 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778242 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 00:01:52.782758 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778245 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 00:01:52.782758 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778247 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 00:01:52.782758 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778250 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 00:01:52.782758 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778252 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 00:01:52.782758 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778254 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 00:01:52.782758 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778257 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 00:01:52.782758 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778259 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 00:01:52.782758 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778262 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 00:01:52.782758 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778264 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 00:01:52.782758 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778267 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 00:01:52.782758 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778270 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 00:01:52.782758 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778273 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 00:01:52.782758 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778276 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 00:01:52.782758 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778278 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 00:01:52.782758 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778280 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 00:01:52.782758 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778284 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 00:01:52.782758 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778289 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 00:01:52.782758 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778293 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 00:01:52.782758 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778297 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 00:01:52.782758 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778323 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 00:01:52.783304 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778343 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 00:01:52.783304 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778347 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 00:01:52.783304 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778350 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 00:01:52.783304 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778354 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 00:01:52.783304 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778357 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 00:01:52.783304 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778360 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 00:01:52.783304 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778363 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 00:01:52.783304 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778366 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 00:01:52.783304 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778369 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 00:01:52.783304 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778371 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 00:01:52.783304 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778374 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 00:01:52.783304 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778377 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 00:01:52.783304 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778380 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 00:01:52.783304 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778383 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 00:01:52.783304 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778385 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 00:01:52.783304 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778388 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 00:01:52.783304 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778391 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 00:01:52.783304 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778394 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 00:01:52.783304 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778397 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 00:01:52.783781 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778399 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 00:01:52.783781 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778402 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 00:01:52.783781 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778404 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 00:01:52.783781 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778407 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 00:01:52.783781 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778409 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 00:01:52.783781 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778412 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 00:01:52.783781 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778415 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 00:01:52.783781 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778418 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 00:01:52.783781 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.778420 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 00:01:52.783781 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778502 2572 flags.go:64] FLAG: --address="0.0.0.0" Jul 30 00:01:52.783781 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778509 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 30 00:01:52.783781 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778516 2572 flags.go:64] FLAG: --anonymous-auth="true" Jul 30 00:01:52.783781 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778521 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 30 00:01:52.783781 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778526 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 30 00:01:52.783781 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778531 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 30 00:01:52.783781 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778535 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 30 00:01:52.783781 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778539 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 30 00:01:52.783781 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778543 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 30 00:01:52.783781 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778546 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 30 00:01:52.783781 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778549 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 30 00:01:52.783781 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778553 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 30 00:01:52.784326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778556 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 30 00:01:52.784326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778559 2572 flags.go:64] FLAG: --cgroup-root="" Jul 30 00:01:52.784326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778562 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 30 00:01:52.784326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778565 2572 flags.go:64] FLAG: --client-ca-file="" Jul 30 00:01:52.784326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778568 2572 flags.go:64] FLAG: --cloud-provider="external" Jul 30 00:01:52.784326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778571 2572 flags.go:64] FLAG: --cluster-dns="[]" Jul 30 00:01:52.784326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778575 2572 flags.go:64] FLAG: --cluster-domain="" Jul 30 00:01:52.784326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778578 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 30 00:01:52.784326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778582 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 30 00:01:52.784326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778585 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 30 00:01:52.784326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778588 2572 flags.go:64] FLAG: --container-log-max-files="5" Jul 30 00:01:52.784326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778592 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 30 00:01:52.784326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778596 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 30 00:01:52.784326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778599 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 30 00:01:52.784326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778602 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 30 00:01:52.784326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778605 2572 flags.go:64] FLAG: --contention-profiling="false" Jul 30 00:01:52.784326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778609 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 30 00:01:52.784326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778612 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 30 00:01:52.784326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778615 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 30 00:01:52.784326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778618 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 30 00:01:52.784326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778622 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 30 00:01:52.784326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778625 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 30 00:01:52.784326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778628 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 30 00:01:52.784326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778631 2572 flags.go:64] FLAG: --enable-load-reader="false" Jul 30 00:01:52.784326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778636 2572 flags.go:64] FLAG: --enable-server="true" Jul 30 00:01:52.784931 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778639 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 30 00:01:52.784931 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778645 2572 flags.go:64] FLAG: --event-burst="100" Jul 30 00:01:52.784931 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778648 2572 flags.go:64] FLAG: --event-qps="50" Jul 30 00:01:52.784931 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778651 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 30 00:01:52.784931 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778654 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 30 00:01:52.784931 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778658 2572 flags.go:64] FLAG: --eviction-hard="" Jul 30 00:01:52.784931 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778661 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 30 00:01:52.784931 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778664 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 30 00:01:52.784931 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778667 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 30 00:01:52.784931 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778671 2572 flags.go:64] FLAG: --eviction-soft="" Jul 30 00:01:52.784931 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778673 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 30 00:01:52.784931 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778676 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 30 00:01:52.784931 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778679 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 30 00:01:52.784931 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778682 2572 flags.go:64] FLAG: --experimental-mounter-path="" Jul 30 00:01:52.784931 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778685 2572 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 30 00:01:52.784931 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778688 2572 flags.go:64] FLAG: --fail-swap-on="true" Jul 30 00:01:52.784931 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778691 2572 flags.go:64] FLAG: --feature-gates="" Jul 30 00:01:52.784931 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778694 2572 flags.go:64] FLAG: --file-check-frequency="20s" Jul 30 00:01:52.784931 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778698 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 30 00:01:52.784931 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778701 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 30 00:01:52.784931 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778704 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 30 00:01:52.784931 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778707 2572 flags.go:64] FLAG: --healthz-port="10248" Jul 30 00:01:52.784931 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778712 2572 flags.go:64] FLAG: --help="false" Jul 30 00:01:52.784931 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778716 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-135-118.ec2.internal" Jul 30 00:01:52.784931 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778719 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 30 00:01:52.785633 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778722 2572 flags.go:64] FLAG: --http-check-frequency="20s" Jul 30 00:01:52.785633 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778725 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 30 00:01:52.785633 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778728 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 30 00:01:52.785633 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778732 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 30 00:01:52.785633 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778735 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 30 00:01:52.785633 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778738 2572 flags.go:64] FLAG: --image-service-endpoint="" Jul 30 00:01:52.785633 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778741 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 30 00:01:52.785633 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778744 2572 flags.go:64] FLAG: --kube-api-burst="100" Jul 30 00:01:52.785633 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778747 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 30 00:01:52.785633 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778751 2572 flags.go:64] FLAG: --kube-api-qps="50" Jul 30 00:01:52.785633 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778754 2572 flags.go:64] FLAG: --kube-reserved="" Jul 30 00:01:52.785633 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778757 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 30 00:01:52.785633 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778760 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 30 00:01:52.785633 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778763 2572 flags.go:64] FLAG: --kubelet-cgroups="" Jul 30 00:01:52.785633 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778766 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 30 00:01:52.785633 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778769 2572 flags.go:64] FLAG: --lock-file="" Jul 30 00:01:52.785633 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778772 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 30 00:01:52.785633 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778775 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 30 00:01:52.785633 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778779 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 30 00:01:52.785633 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778784 2572 flags.go:64] FLAG: --log-json-split-stream="false" Jul 30 00:01:52.785633 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778788 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 30 00:01:52.785633 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778790 2572 flags.go:64] FLAG: --log-text-split-stream="false" Jul 30 00:01:52.785633 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778793 2572 flags.go:64] FLAG: --logging-format="text" Jul 30 00:01:52.786205 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778796 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 30 00:01:52.786205 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778800 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 30 00:01:52.786205 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778803 2572 flags.go:64] FLAG: --manifest-url="" Jul 30 00:01:52.786205 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778806 2572 flags.go:64] FLAG: --manifest-url-header="" Jul 30 00:01:52.786205 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778810 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 30 00:01:52.786205 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778813 2572 flags.go:64] FLAG: --max-open-files="1000000" Jul 30 00:01:52.786205 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778818 2572 flags.go:64] FLAG: --max-pods="110" Jul 30 00:01:52.786205 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778821 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 30 00:01:52.786205 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778824 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 30 00:01:52.786205 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778827 2572 flags.go:64] FLAG: --memory-manager-policy="None" Jul 30 00:01:52.786205 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778830 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 30 00:01:52.786205 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778833 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 30 00:01:52.786205 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778836 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 30 00:01:52.786205 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778839 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 30 00:01:52.786205 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778848 2572 flags.go:64] FLAG: --node-status-max-images="50" Jul 30 00:01:52.786205 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778851 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 30 00:01:52.786205 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778854 2572 flags.go:64] FLAG: --oom-score-adj="-999" Jul 30 00:01:52.786205 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778857 2572 flags.go:64] FLAG: --pod-cidr="" Jul 30 00:01:52.786205 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778860 2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 30 00:01:52.786205 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778866 2572 flags.go:64] FLAG: --pod-manifest-path="" Jul 30 00:01:52.786205 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778869 2572 flags.go:64] FLAG: --pod-max-pids="-1" Jul 30 00:01:52.786205 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778872 2572 flags.go:64] FLAG: --pods-per-core="0" Jul 30 00:01:52.786205 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778875 2572 flags.go:64] FLAG: --port="10250" Jul 30 00:01:52.786205 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778878 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 30 00:01:52.786837 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778881 2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cc0f82dd62c577ef" Jul 30 00:01:52.786837 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778884 2572 flags.go:64] FLAG: --qos-reserved="" Jul 30 00:01:52.786837 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778887 2572 flags.go:64] FLAG: --read-only-port="10255" Jul 30 00:01:52.786837 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778890 2572 flags.go:64] FLAG: --register-node="true" Jul 30 00:01:52.786837 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778892 2572 flags.go:64] FLAG: --register-with-taints="" Jul 30 00:01:52.786837 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778896 2572 flags.go:64] FLAG: --registry-burst="10" Jul 30 00:01:52.786837 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778899 2572 flags.go:64] FLAG: --registry-qps="5" Jul 30 00:01:52.786837 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778902 2572 flags.go:64] FLAG: --reserved-cpus="" Jul 30 00:01:52.786837 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778904 2572 flags.go:64] FLAG: --reserved-memory="" Jul 30 00:01:52.786837 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778908 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 30 00:01:52.786837 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778911 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 30 00:01:52.786837 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778914 2572 flags.go:64] FLAG: --rotate-certificates="false" Jul 30 00:01:52.786837 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778917 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 30 00:01:52.786837 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778920 2572 flags.go:64] FLAG: --runonce="false" Jul 30 00:01:52.786837 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778922 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 30 00:01:52.786837 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778926 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 30 00:01:52.786837 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778929 2572 flags.go:64] FLAG: --seccomp-default="false" Jul 30 00:01:52.786837 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778932 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 30 00:01:52.786837 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778935 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 30 00:01:52.786837 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778938 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 30 00:01:52.786837 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778941 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 30 00:01:52.786837 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778944 2572 flags.go:64] FLAG: --storage-driver-password="root" Jul 30 00:01:52.786837 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778947 2572 flags.go:64] FLAG: --storage-driver-secure="false" Jul 30 00:01:52.786837 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778950 2572 flags.go:64] FLAG: --storage-driver-table="stats" Jul 30 00:01:52.786837 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778953 2572 flags.go:64] FLAG: --storage-driver-user="root" Jul 30 00:01:52.787482 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778956 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 30 00:01:52.787482 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778959 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 30 00:01:52.787482 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778962 2572 flags.go:64] FLAG: --system-cgroups="" Jul 30 00:01:52.787482 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778966 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 30 00:01:52.787482 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778972 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 30 00:01:52.787482 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778975 2572 flags.go:64] FLAG: --tls-cert-file="" Jul 30 00:01:52.787482 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778978 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 30 00:01:52.787482 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778982 2572 flags.go:64] FLAG: --tls-min-version="" Jul 30 00:01:52.787482 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778985 2572 flags.go:64] FLAG: --tls-private-key-file="" Jul 30 00:01:52.787482 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778988 2572 flags.go:64] FLAG: --topology-manager-policy="none" Jul 30 00:01:52.787482 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778991 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 30 00:01:52.787482 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778994 2572 flags.go:64] FLAG: --topology-manager-scope="container" Jul 30 00:01:52.787482 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.778997 2572 flags.go:64] FLAG: --v="2" Jul 30 00:01:52.787482 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.779001 2572 flags.go:64] FLAG: --version="false" Jul 30 00:01:52.787482 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.779005 2572 flags.go:64] FLAG: --vmodule="" Jul 30 00:01:52.787482 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.779009 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 30 00:01:52.787482 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.779012 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 30 00:01:52.787482 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779144 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 00:01:52.787482 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779149 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 00:01:52.787482 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779153 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 00:01:52.787482 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779158 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 00:01:52.787482 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779161 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 00:01:52.787482 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779164 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 00:01:52.788060 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779167 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 00:01:52.788060 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779169 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 00:01:52.788060 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779172 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 00:01:52.788060 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779175 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 00:01:52.788060 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779178 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 00:01:52.788060 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779180 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 00:01:52.788060 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779183 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 00:01:52.788060 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779186 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 00:01:52.788060 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779189 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 00:01:52.788060 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779192 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 00:01:52.788060 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779194 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 00:01:52.788060 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779197 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 00:01:52.788060 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779200 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 00:01:52.788060 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779203 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 00:01:52.788060 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779206 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 00:01:52.788060 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779208 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 00:01:52.788060 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779211 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 00:01:52.788060 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779213 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 00:01:52.788060 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779216 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 00:01:52.788060 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779218 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 00:01:52.788577 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779221 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 00:01:52.788577 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779224 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 00:01:52.788577 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779226 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 00:01:52.788577 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779229 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 00:01:52.788577 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779231 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 00:01:52.788577 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779234 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 00:01:52.788577 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779237 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 00:01:52.788577 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779239 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 00:01:52.788577 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779242 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 00:01:52.788577 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779244 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 00:01:52.788577 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779247 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 00:01:52.788577 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779250 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 00:01:52.788577 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779252 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 00:01:52.788577 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779255 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 00:01:52.788577 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779257 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 00:01:52.788577 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779260 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 00:01:52.788577 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779263 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 00:01:52.788577 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779266 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 00:01:52.788577 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779268 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 00:01:52.788577 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779271 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 00:01:52.789077 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779274 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 00:01:52.789077 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779276 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 00:01:52.789077 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779279 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 00:01:52.789077 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779282 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 00:01:52.789077 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779285 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 00:01:52.789077 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779288 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 00:01:52.789077 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779291 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 00:01:52.789077 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779293 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 00:01:52.789077 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779296 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 00:01:52.789077 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779298 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 00:01:52.789077 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779302 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 00:01:52.789077 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779306 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 00:01:52.789077 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779309 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 00:01:52.789077 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779311 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 00:01:52.789077 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779314 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 00:01:52.789077 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779317 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 00:01:52.789077 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779320 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 00:01:52.789077 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779322 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 00:01:52.789077 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779325 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 00:01:52.789668 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779328 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 00:01:52.789668 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779331 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 00:01:52.789668 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779335 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 30 00:01:52.789668 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779337 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 00:01:52.789668 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779342 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 00:01:52.789668 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779345 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 00:01:52.789668 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779347 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 00:01:52.789668 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779350 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 00:01:52.789668 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779353 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 00:01:52.789668 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779355 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 00:01:52.789668 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779358 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 00:01:52.789668 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779361 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 00:01:52.789668 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779363 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 00:01:52.789668 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779366 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 00:01:52.789668 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779369 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 00:01:52.789668 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779372 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 00:01:52.789668 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779374 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 00:01:52.789668 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779377 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 00:01:52.789668 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779380 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 00:01:52.790208 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779382 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 00:01:52.790208 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779385 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 00:01:52.790208 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779388 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 00:01:52.790208 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779391 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 00:01:52.790208 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779393 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 00:01:52.790208 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779396 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 00:01:52.790208 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779399 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 00:01:52.790208 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779401 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 00:01:52.790208 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779404 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 00:01:52.790208 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779406 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 00:01:52.790208 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779409 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 00:01:52.790208 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779411 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 00:01:52.790208 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779414 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 00:01:52.790208 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779416 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 00:01:52.790208 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.779419 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 00:01:52.790701 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.780377 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 00:01:52.790701 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.786842 2572 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 30 00:01:52.790701 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.786860 2572 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 30 00:01:52.790701 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.787627 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 30 00:01:52.790701 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.787638 2572 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 30 00:01:52.790701 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787702 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 00:01:52.790701 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787707 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 00:01:52.790701 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787710 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 00:01:52.790701 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787713 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 00:01:52.790701 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787717 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 00:01:52.790701 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787720 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 00:01:52.790701 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787722 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 00:01:52.790701 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787725 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 00:01:52.790701 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787728 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 00:01:52.790701 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787731 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 00:01:52.791094 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787733 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 00:01:52.791094 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787736 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 00:01:52.791094 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787739 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 00:01:52.791094 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787741 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 00:01:52.791094 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787744 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 00:01:52.791094 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787746 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 00:01:52.791094 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787749 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 00:01:52.791094 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787751 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 00:01:52.791094 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787754 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 00:01:52.791094 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787757 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 00:01:52.791094 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787759 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 00:01:52.791094 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787762 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 00:01:52.791094 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787764 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 00:01:52.791094 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787767 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 00:01:52.791094 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787769 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 00:01:52.791094 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787772 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 00:01:52.791094 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787775 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 00:01:52.791094 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787778 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 00:01:52.791094 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787781 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 00:01:52.791094 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787786 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 00:01:52.791611 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787791 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 00:01:52.791611 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787795 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 00:01:52.791611 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787799 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 00:01:52.791611 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787802 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 00:01:52.791611 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787805 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 00:01:52.791611 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787808 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 00:01:52.791611 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787810 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 00:01:52.791611 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787813 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 00:01:52.791611 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787815 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 00:01:52.791611 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787818 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 00:01:52.791611 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787821 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 00:01:52.791611 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787823 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 00:01:52.791611 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787826 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 00:01:52.791611 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787828 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 00:01:52.791611 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787831 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 00:01:52.791611 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787835 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 00:01:52.791611 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787838 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 00:01:52.791611 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787841 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 00:01:52.791611 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787844 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 00:01:52.792091 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787847 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 00:01:52.792091 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787850 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 30 00:01:52.792091 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787852 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 00:01:52.792091 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787855 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 00:01:52.792091 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787858 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 00:01:52.792091 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787861 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 00:01:52.792091 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787864 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 00:01:52.792091 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787867 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 00:01:52.792091 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787870 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 00:01:52.792091 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787872 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 00:01:52.792091 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787875 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 00:01:52.792091 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787878 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 00:01:52.792091 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787882 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 00:01:52.792091 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787886 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 00:01:52.792091 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787888 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 00:01:52.792091 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787891 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 00:01:52.792091 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787894 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 00:01:52.792091 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787897 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 00:01:52.792091 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787899 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 00:01:52.792091 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787902 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 00:01:52.792636 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787904 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 00:01:52.792636 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787907 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 00:01:52.792636 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787909 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 00:01:52.792636 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787912 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 00:01:52.792636 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787914 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 00:01:52.792636 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787917 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 00:01:52.792636 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787919 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 00:01:52.792636 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787922 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 00:01:52.792636 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787924 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 00:01:52.792636 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787927 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 00:01:52.792636 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787930 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 00:01:52.792636 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787932 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 00:01:52.792636 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787935 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 00:01:52.792636 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787937 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 00:01:52.792636 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787940 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 00:01:52.792636 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787943 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 00:01:52.792636 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787945 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 00:01:52.792636 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787947 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 00:01:52.792636 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787950 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 00:01:52.793206 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787952 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 00:01:52.793206 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787955 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 00:01:52.793206 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787957 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 00:01:52.793206 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787960 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 00:01:52.793206 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787963 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 00:01:52.793206 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787966 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 00:01:52.793206 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787970 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 00:01:52.793206 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787973 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 00:01:52.793206 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787975 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 00:01:52.793206 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787978 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 00:01:52.793206 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.787981 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 00:01:52.793206 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.787986 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 00:01:52.793206 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788099 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 00:01:52.793206 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788103 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 00:01:52.793206 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788117 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 00:01:52.793587 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788120 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 00:01:52.793587 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788123 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 00:01:52.793587 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788126 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 00:01:52.793587 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788128 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 00:01:52.793587 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788131 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 00:01:52.793587 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788134 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 00:01:52.793587 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788138 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 00:01:52.793587 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788142 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 00:01:52.793587 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788145 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 00:01:52.793587 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788148 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 00:01:52.793587 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788151 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 00:01:52.793587 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788154 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 00:01:52.793587 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788157 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 00:01:52.793587 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788161 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 00:01:52.793587 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788165 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 00:01:52.793587 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788167 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 00:01:52.793587 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788170 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 00:01:52.793587 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788173 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 00:01:52.793587 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788175 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 00:01:52.794056 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788178 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 00:01:52.794056 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788180 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 00:01:52.794056 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788183 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 00:01:52.794056 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788185 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 00:01:52.794056 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788188 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 00:01:52.794056 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788192 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 00:01:52.794056 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788195 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 00:01:52.794056 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788198 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 00:01:52.794056 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788201 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 00:01:52.794056 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788204 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 00:01:52.794056 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788206 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 00:01:52.794056 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788209 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 00:01:52.794056 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788211 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 00:01:52.794056 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788214 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 30 00:01:52.794056 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788217 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 00:01:52.794056 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788219 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 00:01:52.794056 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788221 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 00:01:52.794056 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788224 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 00:01:52.794056 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788227 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 00:01:52.794536 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788229 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 00:01:52.794536 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788232 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 00:01:52.794536 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788234 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 00:01:52.794536 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788237 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 00:01:52.794536 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788239 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 00:01:52.794536 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788242 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 00:01:52.794536 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788245 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 00:01:52.794536 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788247 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 00:01:52.794536 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788250 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 00:01:52.794536 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788252 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 00:01:52.794536 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788255 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 00:01:52.794536 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788257 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 00:01:52.794536 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788260 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 00:01:52.794536 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788262 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 00:01:52.794536 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788265 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 00:01:52.794536 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788267 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 00:01:52.794536 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788270 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 00:01:52.794536 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788273 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 00:01:52.794536 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788276 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 00:01:52.794536 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788278 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 00:01:52.795037 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788281 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 00:01:52.795037 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788284 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 00:01:52.795037 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788286 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 00:01:52.795037 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788289 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 00:01:52.795037 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788291 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 00:01:52.795037 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788294 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 00:01:52.795037 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788296 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 00:01:52.795037 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788299 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 00:01:52.795037 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788301 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 00:01:52.795037 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788303 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 00:01:52.795037 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788306 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 00:01:52.795037 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788308 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 00:01:52.795037 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788311 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 00:01:52.795037 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788314 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 00:01:52.795037 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788316 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 00:01:52.795037 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788318 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 00:01:52.795037 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788321 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 00:01:52.795037 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788324 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 00:01:52.795037 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788326 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 00:01:52.795037 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788329 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 00:01:52.795526 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788331 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 00:01:52.795526 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788334 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 00:01:52.795526 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788336 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 00:01:52.795526 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788339 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 00:01:52.795526 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788341 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 00:01:52.795526 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788343 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 00:01:52.795526 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788346 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 00:01:52.795526 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788348 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 00:01:52.795526 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788351 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 00:01:52.795526 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788353 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 00:01:52.795526 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788356 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 00:01:52.795526 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788359 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 00:01:52.795526 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788361 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 00:01:52.795526 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788364 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 00:01:52.795526 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788367 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 00:01:52.795526 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788369 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 00:01:52.795526 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788372 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 00:01:52.795526 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:52.788374 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 00:01:52.795994 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.788380 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 00:01:52.795994 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.788517 2572 server.go:962] "Client rotation is on, will bootstrap in background" Jul 30 00:01:52.795994 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.792000 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 30 00:01:52.795994 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.793469 2572 server.go:1019] "Starting client certificate rotation" Jul 30 00:01:52.795994 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.793571 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 30 00:01:52.795994 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.793601 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 30 00:01:52.795994 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.795882 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 30 00:01:52.797072 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.797048 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 30 00:01:52.811657 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.811628 2572 log.go:25] "Validated CRI v1 runtime API" Jul 30 00:01:52.813620 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.813606 2572 log.go:25] "Validated CRI v1 image API" Jul 30 00:01:52.814477 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.814459 2572 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 30 00:01:52.816796 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.816776 2572 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 975ddafc-3e1b-4692-b51c-adb0b8b28f12:/dev/nvme0n1p4 99e676b3-9c42-4007-9877-e4ace8f47ca5:/dev/nvme0n1p3] Jul 30 00:01:52.816873 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.816794 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 30 00:01:52.821622 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.821605 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 30 00:01:52.822782 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.822650 2572 manager.go:216] Machine: {Timestamp:2026-07-30 00:01:52.821352699 +0000 UTC m=+0.324610951 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100323 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cb16b639bc2c214046e2ff9ab829e SystemUUID:ec2cb16b-639b-c2c2-1404-6e2ff9ab829e BootID:e4e5be5d-4fc3-4147-8498-5d6e79401bb3 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e8:e0:b0:e9:5b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e8:e0:b0:e9:5b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:c4:8b:7e:98:7a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 30 00:01:52.822782 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.822780 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 30 00:01:52.822899 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.822865 2572 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 30 00:01:52.825807 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.825785 2572 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 30 00:01:52.826390 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.825920 2572 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-118.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 30 00:01:52.826953 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.826873 2572 topology_manager.go:138] "Creating topology manager with none policy" Jul 30 00:01:52.826953 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.826894 2572 container_manager_linux.go:309] "Creating device plugin manager" Jul 30 00:01:52.826953 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.826906 2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 30 00:01:52.828090 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.827486 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 30 00:01:52.829436 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.829424 2572 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 30 00:01:52.832115 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.832093 2572 state_mem.go:36] "Initialized new in-memory state store" Jul 30 00:01:52.832251 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.832241 2572 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 30 00:01:52.832285 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.832272 2572 kubelet.go:486] "Attempting to sync node with API server" Jul 30 00:01:52.832285 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.832283 2572 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 30 00:01:52.832338 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.832294 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 30 00:01:52.832338 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.832302 2572 kubelet.go:398] "Adding apiserver pod source" Jul 30 00:01:52.832338 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.832311 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 30 00:01:52.832682 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.832664 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 30 00:01:52.832682 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.832685 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 30 00:01:52.834771 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.834755 2572 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 30 00:01:52.836710 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.836696 2572 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 30 00:01:52.836754 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.836726 2572 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 30 00:01:52.838217 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.838204 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 30 00:01:52.838280 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.838222 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 30 00:01:52.838280 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.838228 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 30 00:01:52.838280 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.838234 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 30 00:01:52.838280 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.838243 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 30 00:01:52.838280 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.838248 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 30 00:01:52.838280 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.838257 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 30 00:01:52.838280 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.838262 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 30 00:01:52.838280 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.838269 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 30 00:01:52.838280 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.838274 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 30 00:01:52.838280 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.838283 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 30 00:01:52.838576 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.838289 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 30 00:01:52.838977 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.838967 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 30 00:01:52.838977 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.838977 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 30 00:01:52.841969 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.841946 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-118.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 30 00:01:52.842264 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.842247 2572 server.go:1268] "Started kubelet" Jul 30 00:01:52.842426 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.842382 2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 30 00:01:52.842538 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.842505 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 30 00:01:52.842573 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.842554 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 30 00:01:52.843790 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:52.843761 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-118.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 30 00:01:52.843943 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:52.843900 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 30 00:01:52.844394 ip-10-0-135-118 systemd[1]: Started Kubernetes Kubelet. Jul 30 00:01:52.845510 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.845171 2572 server.go:310] "Adding debug handlers to kubelet server" Jul 30 00:01:52.845822 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.845803 2572 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 30 00:01:52.848031 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:52.847768 2572 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 30 00:01:52.848031 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:52.846973 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-118.ec2.internal.18c6e84b66b3a86c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-118.ec2.internal,UID:ip-10-0-135-118.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-118.ec2.internal,},FirstTimestamp:2026-07-30 00:01:52.842221676 +0000 UTC m=+0.345479926,LastTimestamp:2026-07-30 00:01:52.842221676 +0000 UTC m=+0.345479926,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-118.ec2.internal,}" Jul 30 00:01:52.853589 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.853573 2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 30 00:01:52.853683 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.853591 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 30 00:01:52.853744 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:52.853687 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-118.ec2.internal\" not found" Jul 30 00:01:52.854103 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.854082 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 30 00:01:52.854103 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.854082 2572 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 30 00:01:52.854257 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.854128 2572 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 30 00:01:52.854257 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.854213 2572 reconstruct.go:97] "Volume reconstruction finished" Jul 30 00:01:52.854257 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.854221 2572 reconciler.go:29] "Reconciler: start to sync state" Jul 30 00:01:52.854421 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.854403 2572 factory.go:55] Registering systemd factory Jul 30 00:01:52.854421 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.854422 2572 factory.go:223] Registration of the systemd container factory successfully Jul 30 00:01:52.854688 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.854665 2572 factory.go:153] Registering CRI-O factory Jul 30 00:01:52.854688 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.854685 2572 factory.go:223] Registration of the crio container factory successfully Jul 30 00:01:52.854793 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.854738 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 30 00:01:52.854793 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.854766 2572 factory.go:103] Registering Raw factory Jul 30 00:01:52.854793 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.854780 2572 manager.go:1196] Started watching for new ooms in manager Jul 30 00:01:52.855202 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.855187 2572 manager.go:349] Starting recovery of all containers Jul 30 00:01:52.864089 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:52.864066 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 30 00:01:52.864192 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:52.864087 2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-118.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 30 00:01:52.867501 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.867484 2572 manager.go:354] Recovery completed Jul 30 00:01:52.872934 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.872921 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 00:01:52.875818 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.875792 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-118.ec2.internal" event="NodeHasSufficientMemory" Jul 30 00:01:52.875937 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.875836 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-118.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 00:01:52.875937 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.875851 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-118.ec2.internal" event="NodeHasSufficientPID" Jul 30 00:01:52.876531 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.876516 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 00:01:52.876602 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.876515 2572 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 30 00:01:52.876602 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.876549 2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 30 00:01:52.876602 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.876568 2572 state_mem.go:36] "Initialized new in-memory state store" Jul 30 00:01:52.877191 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.877174 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-118.ec2.internal" event="NodeHasSufficientMemory" Jul 30 00:01:52.877271 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.877208 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-118.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 00:01:52.877271 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.877222 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-118.ec2.internal" event="NodeHasSufficientPID" Jul 30 00:01:52.877887 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.877873 2572 policy_none.go:49] "None policy: Start" Jul 30 00:01:52.877936 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.877895 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 30 00:01:52.877936 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.877911 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 30 00:01:52.879222 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.879207 2572 policy_none.go:47] "Start" Jul 30 00:01:52.885999 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:52.885926 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-118.ec2.internal.18c6e84b68b449b8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-118.ec2.internal,UID:ip-10-0-135-118.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-118.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-118.ec2.internal,},FirstTimestamp:2026-07-30 00:01:52.8758174 +0000 UTC m=+0.379075651,LastTimestamp:2026-07-30 00:01:52.8758174 +0000 UTC m=+0.379075651,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-118.ec2.internal,}" Jul 30 00:01:52.891621 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:52.891518 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-118.ec2.internal.18c6e84b68b4af55 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-118.ec2.internal,UID:ip-10-0-135-118.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-118.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-118.ec2.internal,},FirstTimestamp:2026-07-30 00:01:52.875843413 +0000 UTC m=+0.379101665,LastTimestamp:2026-07-30 00:01:52.875843413 +0000 UTC m=+0.379101665,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-118.ec2.internal,}" Jul 30 00:01:52.895228 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:52.895140 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-118.ec2.internal.18c6e84b68b4e786 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-118.ec2.internal,UID:ip-10-0-135-118.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-118.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-118.ec2.internal,},FirstTimestamp:2026-07-30 00:01:52.875857798 +0000 UTC m=+0.379116049,LastTimestamp:2026-07-30 00:01:52.875857798 +0000 UTC m=+0.379116049,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-118.ec2.internal,}" Jul 30 00:01:52.903718 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:52.903604 2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-118.ec2.internal.18c6e84b68b449b8\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-118.ec2.internal.18c6e84b68b449b8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-118.ec2.internal,UID:ip-10-0-135-118.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-118.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-118.ec2.internal,},FirstTimestamp:2026-07-30 00:01:52.8758174 +0000 UTC m=+0.379075651,LastTimestamp:2026-07-30 00:01:52.877190137 +0000 UTC m=+0.380448392,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-118.ec2.internal,}" Jul 30 00:01:52.909892 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.909865 2572 manager.go:337] "Starting Device Plugin manager" Jul 30 00:01:52.921488 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:52.909922 2572 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 30 00:01:52.921488 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.909937 2572 server.go:87] "Starting device plugin registration server" Jul 30 00:01:52.921488 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.910275 2572 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 30 00:01:52.921488 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.910288 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 30 00:01:52.921488 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.910451 2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 30 00:01:52.921488 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.910519 2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 30 00:01:52.921488 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.910527 2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 30 00:01:52.921488 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:52.911142 2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 30 00:01:52.921488 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:52.911202 2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-118.ec2.internal\" not found" Jul 30 00:01:52.921488 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:52.913073 2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-118.ec2.internal.18c6e84b68b4af55\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-118.ec2.internal.18c6e84b68b4af55 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-118.ec2.internal,UID:ip-10-0-135-118.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-118.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-118.ec2.internal,},FirstTimestamp:2026-07-30 00:01:52.875843413 +0000 UTC m=+0.379101665,LastTimestamp:2026-07-30 00:01:52.877214836 +0000 UTC m=+0.380473093,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-118.ec2.internal,}" Jul 30 00:01:52.921488 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.915518 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f9rcp" Jul 30 00:01:52.921488 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:52.921312 2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-118.ec2.internal.18c6e84b68b4e786\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-118.ec2.internal.18c6e84b68b4e786 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-118.ec2.internal,UID:ip-10-0-135-118.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-118.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-118.ec2.internal,},FirstTimestamp:2026-07-30 00:01:52.875857798 +0000 UTC m=+0.379116049,LastTimestamp:2026-07-30 00:01:52.87722767 +0000 UTC m=+0.380485924,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-118.ec2.internal,}" Jul 30 00:01:52.922640 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.922617 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f9rcp" Jul 30 00:01:52.984681 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.984640 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 30 00:01:52.986768 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.986349 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 30 00:01:52.986768 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.986394 2572 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 30 00:01:52.986768 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.986424 2572 kubelet.go:2448] "Starting kubelet main sync loop" Jul 30 00:01:52.986768 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:52.986468 2572 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 30 00:01:52.989920 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:52.989896 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 00:01:53.010848 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.010818 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 00:01:53.011754 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.011736 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-118.ec2.internal" event="NodeHasSufficientMemory" Jul 30 00:01:53.011860 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.011766 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-118.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 00:01:53.011860 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.011781 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-118.ec2.internal" event="NodeHasSufficientPID" Jul 30 00:01:53.011860 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.011837 2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-118.ec2.internal" Jul 30 00:01:53.020495 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.020480 2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-118.ec2.internal" Jul 30 00:01:53.020542 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:53.020504 2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-118.ec2.internal\": node \"ip-10-0-135-118.ec2.internal\" not found" Jul 30 00:01:53.039192 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:53.039173 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-118.ec2.internal\" not found" Jul 30 00:01:53.086719 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.086692 2572 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-118.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-118.ec2.internal"] Jul 30 00:01:53.086847 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.086768 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 00:01:53.087806 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.087784 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-118.ec2.internal" event="NodeHasSufficientMemory" Jul 30 00:01:53.087893 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.087814 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-118.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 00:01:53.087893 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.087828 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-118.ec2.internal" event="NodeHasSufficientPID" Jul 30 00:01:53.089095 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.089082 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 00:01:53.089263 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.089251 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-118.ec2.internal" Jul 30 00:01:53.089298 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.089288 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 00:01:53.089797 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.089780 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-118.ec2.internal" event="NodeHasSufficientMemory" Jul 30 00:01:53.089861 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.089811 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-118.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 00:01:53.089861 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.089822 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-118.ec2.internal" event="NodeHasSufficientPID" Jul 30 00:01:53.089861 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.089786 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-118.ec2.internal" event="NodeHasSufficientMemory" Jul 30 00:01:53.089966 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.089882 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-118.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 00:01:53.089966 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.089896 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-118.ec2.internal" event="NodeHasSufficientPID" Jul 30 00:01:53.091063 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.091049 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-118.ec2.internal" Jul 30 00:01:53.091134 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.091075 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 00:01:53.091725 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.091708 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-118.ec2.internal" event="NodeHasSufficientMemory" Jul 30 00:01:53.091789 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.091735 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-118.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 00:01:53.091789 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.091748 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-118.ec2.internal" event="NodeHasSufficientPID" Jul 30 00:01:53.118958 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:53.118933 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-118.ec2.internal\" not found" node="ip-10-0-135-118.ec2.internal" Jul 30 00:01:53.122369 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:53.122351 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-118.ec2.internal\" not found" node="ip-10-0-135-118.ec2.internal" Jul 30 00:01:53.139334 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:53.139309 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-118.ec2.internal\" not found" Jul 30 00:01:53.156342 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.156314 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7ffdecdbcec9362ce0d8e8d2332c8e14-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-118.ec2.internal\" (UID: \"7ffdecdbcec9362ce0d8e8d2332c8e14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-118.ec2.internal" Jul 30 00:01:53.156412 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.156347 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ffdecdbcec9362ce0d8e8d2332c8e14-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-118.ec2.internal\" (UID: \"7ffdecdbcec9362ce0d8e8d2332c8e14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-118.ec2.internal" Jul 30 00:01:53.156412 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.156368 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/60d5ee2ac122fad92c39ced53363933e-config\") pod \"kube-apiserver-proxy-ip-10-0-135-118.ec2.internal\" (UID: \"60d5ee2ac122fad92c39ced53363933e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-118.ec2.internal" Jul 30 00:01:53.240121 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:53.240064 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-118.ec2.internal\" not found" Jul 30 00:01:53.257502 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.257476 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/60d5ee2ac122fad92c39ced53363933e-config\") pod \"kube-apiserver-proxy-ip-10-0-135-118.ec2.internal\" (UID: \"60d5ee2ac122fad92c39ced53363933e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-118.ec2.internal" Jul 30 00:01:53.257567 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.257494 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/60d5ee2ac122fad92c39ced53363933e-config\") pod \"kube-apiserver-proxy-ip-10-0-135-118.ec2.internal\" (UID: \"60d5ee2ac122fad92c39ced53363933e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-118.ec2.internal" Jul 30 00:01:53.257567 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.257536 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7ffdecdbcec9362ce0d8e8d2332c8e14-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-118.ec2.internal\" (UID: \"7ffdecdbcec9362ce0d8e8d2332c8e14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-118.ec2.internal" Jul 30 00:01:53.257567 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.257555 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ffdecdbcec9362ce0d8e8d2332c8e14-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-118.ec2.internal\" (UID: \"7ffdecdbcec9362ce0d8e8d2332c8e14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-118.ec2.internal" Jul 30 00:01:53.257659 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.257590 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ffdecdbcec9362ce0d8e8d2332c8e14-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-118.ec2.internal\" (UID: \"7ffdecdbcec9362ce0d8e8d2332c8e14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-118.ec2.internal" Jul 30 00:01:53.257659 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.257623 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7ffdecdbcec9362ce0d8e8d2332c8e14-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-118.ec2.internal\" (UID: \"7ffdecdbcec9362ce0d8e8d2332c8e14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-118.ec2.internal" Jul 30 00:01:53.341088 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:53.340874 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-118.ec2.internal\" not found" Jul 30 00:01:53.422468 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.422419 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-118.ec2.internal" Jul 30 00:01:53.424585 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.424565 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-118.ec2.internal" Jul 30 00:01:53.442072 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:53.442046 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-118.ec2.internal\" not found" Jul 30 00:01:53.543010 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:53.542972 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-118.ec2.internal\" not found" Jul 30 00:01:53.649603 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:53.649512 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-118.ec2.internal\" not found" Jul 30 00:01:53.757238 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:53.757191 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-118.ec2.internal\" not found" Jul 30 00:01:53.795428 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.795395 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 30 00:01:53.795862 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.795592 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 30 00:01:53.838973 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.838945 2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 00:01:53.854282 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.854258 2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 00:01:53.926350 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.926237 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 23:56:53 +0000 UTC" deadline="2028-05-01 23:51:11.856116502 +0000 UTC" Jul 30 00:01:53.926350 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.926280 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15407h49m17.929839389s" Jul 30 00:01:53.954481 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.954431 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-118.ec2.internal" Jul 30 00:01:53.971796 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.971766 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 30 00:01:53.972631 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.972614 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-118.ec2.internal" Jul 30 00:01:53.981205 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:53.981186 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 30 00:01:54.078996 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:54.078954 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ffdecdbcec9362ce0d8e8d2332c8e14.slice/crio-c3766633f62b6e5297cff344314d22e4149337b53e839bb23129f9dbe7f967f0 WatchSource:0}: Error finding container c3766633f62b6e5297cff344314d22e4149337b53e839bb23129f9dbe7f967f0: Status 404 returned error can't find the container with id c3766633f62b6e5297cff344314d22e4149337b53e839bb23129f9dbe7f967f0 Jul 30 00:01:54.084176 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.084155 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 00:01:54.348951 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:54.348915 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d5ee2ac122fad92c39ced53363933e.slice/crio-0b7f848887c7773fce56d39590c29e136f2ae33d3691ac82f9523aa3a4224244 WatchSource:0}: Error finding container 0b7f848887c7773fce56d39590c29e136f2ae33d3691ac82f9523aa3a4224244: Status 404 returned error can't find the container with id 0b7f848887c7773fce56d39590c29e136f2ae33d3691ac82f9523aa3a4224244 Jul 30 00:01:54.426515 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.426481 2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 00:01:54.833258 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.833167 2572 apiserver.go:52] "Watching apiserver" Jul 30 00:01:54.843124 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.843067 2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 30 00:01:54.844027 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.843980 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-77hgn","kube-system/konnectivity-agent-g7zvw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg","openshift-dns/node-resolver-zsmpb","openshift-image-registry/node-ca-lbgfr","openshift-multus/multus-additional-cni-plugins-lg8hw","openshift-network-diagnostics/network-check-target-xpx87","openshift-network-operator/iptables-alerter-v5bld","kube-system/kube-apiserver-proxy-ip-10-0-135-118.ec2.internal","openshift-cluster-node-tuning-operator/tuned-652c8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-118.ec2.internal","openshift-multus/multus-r2jkk","openshift-multus/network-metrics-daemon-nk989","openshift-ovn-kubernetes/ovnkube-node-l48hb"] Jul 30 00:01:54.845811 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.845777 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:01:54.845952 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:54.845877 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpx87" podUID="375724f6-f976-4dc0-a53a-a50fca8d270c" Jul 30 00:01:54.847167 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.847143 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:01:54.847281 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:54.847216 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nk989" podUID="a38e0c1c-7133-40ff-9ba4-9a066da5c41e" Jul 30 00:01:54.848721 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.848697 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg" Jul 30 00:01:54.850190 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.850125 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zsmpb" Jul 30 00:01:54.851711 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.851688 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 30 00:01:54.851803 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.851719 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 30 00:01:54.852057 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.852035 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 30 00:01:54.852175 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.852056 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9xd28\"" Jul 30 00:01:54.852399 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.852376 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 30 00:01:54.852732 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.852710 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 30 00:01:54.852824 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.852770 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lbgfr" Jul 30 00:01:54.853310 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.853285 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4hr66\"" Jul 30 00:01:54.853916 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.853895 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 30 00:01:54.854668 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.854650 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lg8hw" Jul 30 00:01:54.855422 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.855401 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 30 00:01:54.855516 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.855440 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lhc52\"" Jul 30 00:01:54.855516 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.855482 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 30 00:01:54.855764 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.855744 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 30 00:01:54.855863 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.855846 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.856905 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.856887 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 30 00:01:54.857372 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.857351 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 30 00:01:54.857472 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.857422 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 30 00:01:54.857472 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.857443 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 30 00:01:54.857588 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.857501 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:01:54.857644 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:54.857581 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77hgn" podUID="46aec834-30bb-4d73-b1b7-f033193e6e99" Jul 30 00:01:54.857730 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.857710 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 30 00:01:54.857850 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.857785 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7mmvs\"" Jul 30 00:01:54.858554 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.858305 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 30 00:01:54.858685 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.858630 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 30 00:01:54.858882 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.858860 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 30 00:01:54.858882 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.858875 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 30 00:01:54.859018 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.858967 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 30 00:01:54.859480 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.859461 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g7zvw" Jul 30 00:01:54.860462 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.860440 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xmr29\"" Jul 30 00:01:54.861343 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.861322 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 30 00:01:54.861453 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.861334 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:54.862080 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.862061 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-fs72l\"" Jul 30 00:01:54.862358 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.862309 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 30 00:01:54.862358 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.862321 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 30 00:01:54.863086 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.863066 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v5bld" Jul 30 00:01:54.864219 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.864146 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 30 00:01:54.864390 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.864372 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 30 00:01:54.864566 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.864552 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hmr2v\"" Jul 30 00:01:54.864943 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.864919 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r2jkk" Jul 30 00:01:54.865440 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.865415 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bf193c2-d465-4e6c-abaf-afc0f08815fa-tmp-dir\") pod \"node-resolver-zsmpb\" (UID: \"6bf193c2-d465-4e6c-abaf-afc0f08815fa\") " pod="openshift-dns/node-resolver-zsmpb" Jul 30 00:01:54.865550 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.865444 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 30 00:01:54.865550 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.865489 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-run-openvswitch\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.865550 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.865521 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-ovnkube-config\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.865685 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.865550 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs\") pod \"network-metrics-daemon-nk989\" (UID: \"a38e0c1c-7133-40ff-9ba4-9a066da5c41e\") " pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:01:54.865685 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.865575 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f48f6e0-a27a-413d-9033-19a19f9a5331-etc-selinux\") pod \"aws-ebs-csi-driver-node-xxsrg\" (UID: \"4f48f6e0-a27a-413d-9033-19a19f9a5331\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg" Jul 30 00:01:54.865685 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.865610 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5bfc194e-138e-4335-87f4-f6837c8f2e49-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lg8hw\" (UID: \"5bfc194e-138e-4335-87f4-f6837c8f2e49\") " pod="openshift-multus/multus-additional-cni-plugins-lg8hw" Jul 30 00:01:54.865685 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.865640 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5bfc194e-138e-4335-87f4-f6837c8f2e49-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lg8hw\" (UID: \"5bfc194e-138e-4335-87f4-f6837c8f2e49\") " pod="openshift-multus/multus-additional-cni-plugins-lg8hw" Jul 30 00:01:54.865685 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.865667 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-run-systemd\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.865859 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.865691 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-etc-openvswitch\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.865859 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.865718 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-host-cni-bin\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.865859 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.865743 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrgck\" (UniqueName: \"kubernetes.io/projected/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-kube-api-access-rrgck\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.865859 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.865748 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 30 00:01:54.865859 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.865768 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bltj\" (UniqueName: \"kubernetes.io/projected/4f48f6e0-a27a-413d-9033-19a19f9a5331-kube-api-access-8bltj\") pod \"aws-ebs-csi-driver-node-xxsrg\" (UID: \"4f48f6e0-a27a-413d-9033-19a19f9a5331\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg" Jul 30 00:01:54.865859 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.865792 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5mhv\" (UniqueName: \"kubernetes.io/projected/375724f6-f976-4dc0-a53a-a50fca8d270c-kube-api-access-m5mhv\") pod \"network-check-target-xpx87\" (UID: \"375724f6-f976-4dc0-a53a-a50fca8d270c\") " pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:01:54.865859 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.865806 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 30 00:01:54.865859 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.865819 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bfc194e-138e-4335-87f4-f6837c8f2e49-cnibin\") pod \"multus-additional-cni-plugins-lg8hw\" (UID: \"5bfc194e-138e-4335-87f4-f6837c8f2e49\") " pod="openshift-multus/multus-additional-cni-plugins-lg8hw" Jul 30 00:01:54.865859 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.865843 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-run-ovn\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.866219 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.865872 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-ovn-node-metrics-cert\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.866219 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.865900 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa41535c-676c-430a-9151-7257622c4076-host\") pod \"node-ca-lbgfr\" (UID: \"aa41535c-676c-430a-9151-7257622c4076\") " pod="openshift-image-registry/node-ca-lbgfr" Jul 30 00:01:54.866219 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.865929 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqgzj\" (UniqueName: \"kubernetes.io/projected/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-kube-api-access-cqgzj\") pod \"network-metrics-daemon-nk989\" (UID: \"a38e0c1c-7133-40ff-9ba4-9a066da5c41e\") " pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:01:54.866219 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.865975 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vglbv\"" Jul 30 00:01:54.866219 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.865984 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f48f6e0-a27a-413d-9033-19a19f9a5331-device-dir\") pod \"aws-ebs-csi-driver-node-xxsrg\" (UID: \"4f48f6e0-a27a-413d-9033-19a19f9a5331\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg" Jul 30 00:01:54.866219 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.866010 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rd44\" (UniqueName: \"kubernetes.io/projected/6bf193c2-d465-4e6c-abaf-afc0f08815fa-kube-api-access-2rd44\") pod \"node-resolver-zsmpb\" (UID: \"6bf193c2-d465-4e6c-abaf-afc0f08815fa\") " pod="openshift-dns/node-resolver-zsmpb" Jul 30 00:01:54.866219 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.866034 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-node-log\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.866219 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.866058 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-host-run-ovn-kubernetes\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.866219 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.866136 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 30 00:01:54.866219 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.866201 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.866687 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.866238 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bfc194e-138e-4335-87f4-f6837c8f2e49-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lg8hw\" (UID: \"5bfc194e-138e-4335-87f4-f6837c8f2e49\") " pod="openshift-multus/multus-additional-cni-plugins-lg8hw" Jul 30 00:01:54.866687 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.866263 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng6wn\" (UniqueName: \"kubernetes.io/projected/5bfc194e-138e-4335-87f4-f6837c8f2e49-kube-api-access-ng6wn\") pod \"multus-additional-cni-plugins-lg8hw\" (UID: \"5bfc194e-138e-4335-87f4-f6837c8f2e49\") " pod="openshift-multus/multus-additional-cni-plugins-lg8hw" Jul 30 00:01:54.866687 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.866288 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f48f6e0-a27a-413d-9033-19a19f9a5331-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xxsrg\" (UID: \"4f48f6e0-a27a-413d-9033-19a19f9a5331\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg" Jul 30 00:01:54.866687 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.866332 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f48f6e0-a27a-413d-9033-19a19f9a5331-socket-dir\") pod \"aws-ebs-csi-driver-node-xxsrg\" (UID: \"4f48f6e0-a27a-413d-9033-19a19f9a5331\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg" Jul 30 00:01:54.866687 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.866379 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f48f6e0-a27a-413d-9033-19a19f9a5331-registration-dir\") pod \"aws-ebs-csi-driver-node-xxsrg\" (UID: \"4f48f6e0-a27a-413d-9033-19a19f9a5331\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg" Jul 30 00:01:54.866687 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.866459 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f48f6e0-a27a-413d-9033-19a19f9a5331-sys-fs\") pod \"aws-ebs-csi-driver-node-xxsrg\" (UID: \"4f48f6e0-a27a-413d-9033-19a19f9a5331\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg" Jul 30 00:01:54.866687 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.866490 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bfc194e-138e-4335-87f4-f6837c8f2e49-cni-binary-copy\") pod \"multus-additional-cni-plugins-lg8hw\" (UID: \"5bfc194e-138e-4335-87f4-f6837c8f2e49\") " pod="openshift-multus/multus-additional-cni-plugins-lg8hw" Jul 30 00:01:54.866687 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.866531 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-host-kubelet\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.866687 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.866558 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-systemd-units\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.866687 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.866604 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-host-run-netns\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.866687 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.866629 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6bf193c2-d465-4e6c-abaf-afc0f08815fa-hosts-file\") pod \"node-resolver-zsmpb\" (UID: \"6bf193c2-d465-4e6c-abaf-afc0f08815fa\") " pod="openshift-dns/node-resolver-zsmpb" Jul 30 00:01:54.866687 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.866653 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5bfc194e-138e-4335-87f4-f6837c8f2e49-os-release\") pod \"multus-additional-cni-plugins-lg8hw\" (UID: \"5bfc194e-138e-4335-87f4-f6837c8f2e49\") " pod="openshift-multus/multus-additional-cni-plugins-lg8hw" Jul 30 00:01:54.866687 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.866677 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-host-slash\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.867345 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.866700 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-var-lib-openvswitch\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.867345 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.866722 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-host-cni-netd\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.867345 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.866773 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-env-overrides\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.867345 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.866874 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5crh\" (UniqueName: \"kubernetes.io/projected/aa41535c-676c-430a-9151-7257622c4076-kube-api-access-l5crh\") pod \"node-ca-lbgfr\" (UID: \"aa41535c-676c-430a-9151-7257622c4076\") " pod="openshift-image-registry/node-ca-lbgfr" Jul 30 00:01:54.867345 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.866915 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bfc194e-138e-4335-87f4-f6837c8f2e49-system-cni-dir\") pod \"multus-additional-cni-plugins-lg8hw\" (UID: \"5bfc194e-138e-4335-87f4-f6837c8f2e49\") " pod="openshift-multus/multus-additional-cni-plugins-lg8hw" Jul 30 00:01:54.867345 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.866941 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-log-socket\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.867345 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.867015 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-ovnkube-script-lib\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.867345 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.867043 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aa41535c-676c-430a-9151-7257622c4076-serviceca\") pod \"node-ca-lbgfr\" (UID: \"aa41535c-676c-430a-9151-7257622c4076\") " pod="openshift-image-registry/node-ca-lbgfr" Jul 30 00:01:54.867795 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.867771 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 30 00:01:54.868101 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.868082 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8s4xs\"" Jul 30 00:01:54.889222 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.889187 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-r5kdf" Jul 30 00:01:54.898535 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.898500 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r5kdf" Jul 30 00:01:54.955965 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.955929 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 30 00:01:54.967657 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.967621 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-var-lib-openvswitch\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.967840 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.967666 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ccd8796b-bcd4-41e9-a946-c43b4c9863cf-host-slash\") pod \"iptables-alerter-v5bld\" (UID: \"ccd8796b-bcd4-41e9-a946-c43b4c9863cf\") " pod="openshift-network-operator/iptables-alerter-v5bld" Jul 30 00:01:54.967840 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.967690 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-etc-sysctl-conf\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:54.967840 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.967707 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-host\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:54.967840 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.967725 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-var-lib-openvswitch\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.967840 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.967733 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6bf193c2-d465-4e6c-abaf-afc0f08815fa-hosts-file\") pod \"node-resolver-zsmpb\" (UID: \"6bf193c2-d465-4e6c-abaf-afc0f08815fa\") " pod="openshift-dns/node-resolver-zsmpb" Jul 30 00:01:54.967840 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.967803 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-host-run-k8s-cni-cncf-io\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:54.967840 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.967813 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6bf193c2-d465-4e6c-abaf-afc0f08815fa-hosts-file\") pod \"node-resolver-zsmpb\" (UID: \"6bf193c2-d465-4e6c-abaf-afc0f08815fa\") " pod="openshift-dns/node-resolver-zsmpb" Jul 30 00:01:54.968258 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.967845 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bfc194e-138e-4335-87f4-f6837c8f2e49-system-cni-dir\") pod \"multus-additional-cni-plugins-lg8hw\" (UID: \"5bfc194e-138e-4335-87f4-f6837c8f2e49\") " pod="openshift-multus/multus-additional-cni-plugins-lg8hw" Jul 30 00:01:54.968258 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.967899 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bfc194e-138e-4335-87f4-f6837c8f2e49-system-cni-dir\") pod \"multus-additional-cni-plugins-lg8hw\" (UID: \"5bfc194e-138e-4335-87f4-f6837c8f2e49\") " pod="openshift-multus/multus-additional-cni-plugins-lg8hw" Jul 30 00:01:54.968258 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.967915 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aa41535c-676c-430a-9151-7257622c4076-serviceca\") pod \"node-ca-lbgfr\" (UID: \"aa41535c-676c-430a-9151-7257622c4076\") " pod="openshift-image-registry/node-ca-lbgfr" Jul 30 00:01:54.968258 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.967959 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bf193c2-d465-4e6c-abaf-afc0f08815fa-tmp-dir\") pod \"node-resolver-zsmpb\" (UID: \"6bf193c2-d465-4e6c-abaf-afc0f08815fa\") " pod="openshift-dns/node-resolver-zsmpb" Jul 30 00:01:54.968258 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.968035 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-run-openvswitch\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.968258 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.968152 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-ovnkube-config\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.968258 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.968177 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-run-openvswitch\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.968258 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.968232 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46aec834-30bb-4d73-b1b7-f033193e6e99-original-pull-secret\") pod \"global-pull-secret-syncer-77hgn\" (UID: \"46aec834-30bb-4d73-b1b7-f033193e6e99\") " pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:01:54.968748 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.968265 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f48f6e0-a27a-413d-9033-19a19f9a5331-etc-selinux\") pod \"aws-ebs-csi-driver-node-xxsrg\" (UID: \"4f48f6e0-a27a-413d-9033-19a19f9a5331\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg" Jul 30 00:01:54.968748 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.968360 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bf193c2-d465-4e6c-abaf-afc0f08815fa-tmp-dir\") pod \"node-resolver-zsmpb\" (UID: \"6bf193c2-d465-4e6c-abaf-afc0f08815fa\") " pod="openshift-dns/node-resolver-zsmpb" Jul 30 00:01:54.968748 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.968389 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4f48f6e0-a27a-413d-9033-19a19f9a5331-etc-selinux\") pod \"aws-ebs-csi-driver-node-xxsrg\" (UID: \"4f48f6e0-a27a-413d-9033-19a19f9a5331\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg" Jul 30 00:01:54.968748 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.968423 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5bfc194e-138e-4335-87f4-f6837c8f2e49-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lg8hw\" (UID: \"5bfc194e-138e-4335-87f4-f6837c8f2e49\") " pod="openshift-multus/multus-additional-cni-plugins-lg8hw" Jul 30 00:01:54.968748 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.968477 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-etc-openvswitch\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.968748 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.968515 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aa41535c-676c-430a-9151-7257622c4076-serviceca\") pod \"node-ca-lbgfr\" (UID: \"aa41535c-676c-430a-9151-7257622c4076\") " pod="openshift-image-registry/node-ca-lbgfr" Jul 30 00:01:54.968748 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.968602 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-host-cni-bin\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.968748 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.968636 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrgck\" (UniqueName: \"kubernetes.io/projected/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-kube-api-access-rrgck\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.968748 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.968690 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-etc-openvswitch\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.968748 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.968744 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7nsm\" (UniqueName: \"kubernetes.io/projected/ccd8796b-bcd4-41e9-a946-c43b4c9863cf-kube-api-access-l7nsm\") pod \"iptables-alerter-v5bld\" (UID: \"ccd8796b-bcd4-41e9-a946-c43b4c9863cf\") " pod="openshift-network-operator/iptables-alerter-v5bld" Jul 30 00:01:54.969129 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.968781 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-host-var-lib-cni-bin\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:54.969129 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.968825 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-host-run-multus-certs\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:54.969129 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.968856 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bltj\" (UniqueName: \"kubernetes.io/projected/4f48f6e0-a27a-413d-9033-19a19f9a5331-kube-api-access-8bltj\") pod \"aws-ebs-csi-driver-node-xxsrg\" (UID: \"4f48f6e0-a27a-413d-9033-19a19f9a5331\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg" Jul 30 00:01:54.969129 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.968878 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-run-ovn\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.969129 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.968895 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f1f125ac-f373-4ad9-8734-263caec79393-agent-certs\") pod \"konnectivity-agent-g7zvw\" (UID: \"f1f125ac-f373-4ad9-8734-263caec79393\") " pod="kube-system/konnectivity-agent-g7zvw" Jul 30 00:01:54.969129 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.968914 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-ovnkube-config\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.969129 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.968971 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-host-cni-bin\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.969129 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.968980 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5bfc194e-138e-4335-87f4-f6837c8f2e49-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lg8hw\" (UID: \"5bfc194e-138e-4335-87f4-f6837c8f2e49\") " pod="openshift-multus/multus-additional-cni-plugins-lg8hw" Jul 30 00:01:54.969129 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969006 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5fgs\" (UniqueName: \"kubernetes.io/projected/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-kube-api-access-f5fgs\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:54.969129 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969064 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-run-ovn\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.969129 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969105 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa41535c-676c-430a-9151-7257622c4076-host\") pod \"node-ca-lbgfr\" (UID: \"aa41535c-676c-430a-9151-7257622c4076\") " pod="openshift-image-registry/node-ca-lbgfr" Jul 30 00:01:54.969129 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969135 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rd44\" (UniqueName: \"kubernetes.io/projected/6bf193c2-d465-4e6c-abaf-afc0f08815fa-kube-api-access-2rd44\") pod \"node-resolver-zsmpb\" (UID: \"6bf193c2-d465-4e6c-abaf-afc0f08815fa\") " pod="openshift-dns/node-resolver-zsmpb" Jul 30 00:01:54.969671 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969159 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa41535c-676c-430a-9151-7257622c4076-host\") pod \"node-ca-lbgfr\" (UID: \"aa41535c-676c-430a-9151-7257622c4076\") " pod="openshift-image-registry/node-ca-lbgfr" Jul 30 00:01:54.969671 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969178 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-host-run-ovn-kubernetes\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.969671 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969273 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.969671 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969283 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-host-run-ovn-kubernetes\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.969671 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969314 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/46aec834-30bb-4d73-b1b7-f033193e6e99-dbus\") pod \"global-pull-secret-syncer-77hgn\" (UID: \"46aec834-30bb-4d73-b1b7-f033193e6e99\") " pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:01:54.969671 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969324 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.969671 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969361 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r79z4\" (UniqueName: \"kubernetes.io/projected/b54c3644-ab98-4033-aab9-9cd95993a4c7-kube-api-access-r79z4\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:54.969671 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969406 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-system-cni-dir\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:54.969671 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969444 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-host-run-netns\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:54.969671 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969467 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bfc194e-138e-4335-87f4-f6837c8f2e49-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lg8hw\" (UID: \"5bfc194e-138e-4335-87f4-f6837c8f2e49\") " pod="openshift-multus/multus-additional-cni-plugins-lg8hw" Jul 30 00:01:54.969671 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969482 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-etc-sysctl-d\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:54.969671 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969506 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-etc-systemd\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:54.969671 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969539 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-cnibin\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:54.969671 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969578 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f48f6e0-a27a-413d-9033-19a19f9a5331-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xxsrg\" (UID: \"4f48f6e0-a27a-413d-9033-19a19f9a5331\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg" Jul 30 00:01:54.969671 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969605 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f48f6e0-a27a-413d-9033-19a19f9a5331-sys-fs\") pod \"aws-ebs-csi-driver-node-xxsrg\" (UID: \"4f48f6e0-a27a-413d-9033-19a19f9a5331\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg" Jul 30 00:01:54.969671 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969628 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bfc194e-138e-4335-87f4-f6837c8f2e49-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lg8hw\" (UID: \"5bfc194e-138e-4335-87f4-f6837c8f2e49\") " pod="openshift-multus/multus-additional-cni-plugins-lg8hw" Jul 30 00:01:54.969671 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969666 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4f48f6e0-a27a-413d-9033-19a19f9a5331-sys-fs\") pod \"aws-ebs-csi-driver-node-xxsrg\" (UID: \"4f48f6e0-a27a-413d-9033-19a19f9a5331\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg" Jul 30 00:01:54.970555 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969689 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bfc194e-138e-4335-87f4-f6837c8f2e49-cni-binary-copy\") pod \"multus-additional-cni-plugins-lg8hw\" (UID: \"5bfc194e-138e-4335-87f4-f6837c8f2e49\") " pod="openshift-multus/multus-additional-cni-plugins-lg8hw" Jul 30 00:01:54.970555 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969724 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-host-kubelet\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.970555 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969725 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4f48f6e0-a27a-413d-9033-19a19f9a5331-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xxsrg\" (UID: \"4f48f6e0-a27a-413d-9033-19a19f9a5331\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg" Jul 30 00:01:54.970555 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969750 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-host-run-netns\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.970555 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969774 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-lib-modules\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:54.970555 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969791 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-host-run-netns\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.970555 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969785 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-host-kubelet\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.970555 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969814 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-var-lib-kubelet\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:54.970555 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969848 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5bfc194e-138e-4335-87f4-f6837c8f2e49-os-release\") pod \"multus-additional-cni-plugins-lg8hw\" (UID: \"5bfc194e-138e-4335-87f4-f6837c8f2e49\") " pod="openshift-multus/multus-additional-cni-plugins-lg8hw" Jul 30 00:01:54.970555 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969873 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-host-slash\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.970555 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969906 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-host-cni-netd\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.970555 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969909 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-host-slash\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.970555 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969927 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5bfc194e-138e-4335-87f4-f6837c8f2e49-os-release\") pod \"multus-additional-cni-plugins-lg8hw\" (UID: \"5bfc194e-138e-4335-87f4-f6837c8f2e49\") " pod="openshift-multus/multus-additional-cni-plugins-lg8hw" Jul 30 00:01:54.970555 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969931 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-env-overrides\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.970555 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969965 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/46aec834-30bb-4d73-b1b7-f033193e6e99-kubelet-config\") pod \"global-pull-secret-syncer-77hgn\" (UID: \"46aec834-30bb-4d73-b1b7-f033193e6e99\") " pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:01:54.970555 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.969981 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-host-cni-netd\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.970555 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970006 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-etc-kubernetes\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:54.971388 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970035 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5crh\" (UniqueName: \"kubernetes.io/projected/aa41535c-676c-430a-9151-7257622c4076-kube-api-access-l5crh\") pod \"node-ca-lbgfr\" (UID: \"aa41535c-676c-430a-9151-7257622c4076\") " pod="openshift-image-registry/node-ca-lbgfr" Jul 30 00:01:54.971388 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970062 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-log-socket\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.971388 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970088 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-ovnkube-script-lib\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.971388 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970113 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-etc-modprobe-d\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:54.971388 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970152 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-multus-daemon-config\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:54.971388 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970156 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-log-socket\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.971388 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970184 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-host-var-lib-cni-multus\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:54.971388 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970210 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs\") pod \"network-metrics-daemon-nk989\" (UID: \"a38e0c1c-7133-40ff-9ba4-9a066da5c41e\") " pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:01:54.971388 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970229 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bfc194e-138e-4335-87f4-f6837c8f2e49-cni-binary-copy\") pod \"multus-additional-cni-plugins-lg8hw\" (UID: \"5bfc194e-138e-4335-87f4-f6837c8f2e49\") " pod="openshift-multus/multus-additional-cni-plugins-lg8hw" Jul 30 00:01:54.971388 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970255 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5bfc194e-138e-4335-87f4-f6837c8f2e49-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lg8hw\" (UID: \"5bfc194e-138e-4335-87f4-f6837c8f2e49\") " pod="openshift-multus/multus-additional-cni-plugins-lg8hw" Jul 30 00:01:54.971388 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970291 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-env-overrides\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.971388 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970296 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-run-systemd\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.971388 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970318 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f1f125ac-f373-4ad9-8734-263caec79393-konnectivity-ca\") pod \"konnectivity-agent-g7zvw\" (UID: \"f1f125ac-f373-4ad9-8734-263caec79393\") " pod="kube-system/konnectivity-agent-g7zvw" Jul 30 00:01:54.971388 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970334 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-run-systemd\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.971388 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970342 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-multus-cni-dir\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:54.971388 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970367 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-multus-socket-dir-parent\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:54.971388 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970385 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-hostroot\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:54.972161 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970399 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-multus-conf-dir\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:54.972161 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:54.970369 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 00:01:54.972161 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970423 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5mhv\" (UniqueName: \"kubernetes.io/projected/375724f6-f976-4dc0-a53a-a50fca8d270c-kube-api-access-m5mhv\") pod \"network-check-target-xpx87\" (UID: \"375724f6-f976-4dc0-a53a-a50fca8d270c\") " pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:01:54.972161 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970464 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bfc194e-138e-4335-87f4-f6837c8f2e49-cnibin\") pod \"multus-additional-cni-plugins-lg8hw\" (UID: \"5bfc194e-138e-4335-87f4-f6837c8f2e49\") " pod="openshift-multus/multus-additional-cni-plugins-lg8hw" Jul 30 00:01:54.972161 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:54.970517 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs podName:a38e0c1c-7133-40ff-9ba4-9a066da5c41e nodeName:}" failed. No retries permitted until 2026-07-30 00:01:55.470475081 +0000 UTC m=+2.973733347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs") pod "network-metrics-daemon-nk989" (UID: "a38e0c1c-7133-40ff-9ba4-9a066da5c41e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 00:01:54.972161 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970580 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-ovn-node-metrics-cert\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.972161 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970586 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bfc194e-138e-4335-87f4-f6837c8f2e49-cnibin\") pod \"multus-additional-cni-plugins-lg8hw\" (UID: \"5bfc194e-138e-4335-87f4-f6837c8f2e49\") " pod="openshift-multus/multus-additional-cni-plugins-lg8hw" Jul 30 00:01:54.972161 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970625 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ccd8796b-bcd4-41e9-a946-c43b4c9863cf-iptables-alerter-script\") pod \"iptables-alerter-v5bld\" (UID: \"ccd8796b-bcd4-41e9-a946-c43b4c9863cf\") " pod="openshift-network-operator/iptables-alerter-v5bld" Jul 30 00:01:54.972161 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970659 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-os-release\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:54.972161 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970694 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqgzj\" (UniqueName: \"kubernetes.io/projected/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-kube-api-access-cqgzj\") pod \"network-metrics-daemon-nk989\" (UID: \"a38e0c1c-7133-40ff-9ba4-9a066da5c41e\") " pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:01:54.972161 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970695 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-ovnkube-script-lib\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.972161 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970722 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f48f6e0-a27a-413d-9033-19a19f9a5331-device-dir\") pod \"aws-ebs-csi-driver-node-xxsrg\" (UID: \"4f48f6e0-a27a-413d-9033-19a19f9a5331\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg" Jul 30 00:01:54.972161 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970793 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4f48f6e0-a27a-413d-9033-19a19f9a5331-device-dir\") pod \"aws-ebs-csi-driver-node-xxsrg\" (UID: \"4f48f6e0-a27a-413d-9033-19a19f9a5331\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg" Jul 30 00:01:54.972161 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970799 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5bfc194e-138e-4335-87f4-f6837c8f2e49-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lg8hw\" (UID: \"5bfc194e-138e-4335-87f4-f6837c8f2e49\") " pod="openshift-multus/multus-additional-cni-plugins-lg8hw" Jul 30 00:01:54.972161 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970837 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-node-log\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.972161 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970866 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-etc-sysconfig\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:54.973020 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970895 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-etc-kubernetes\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:54.973020 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970918 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-run\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:54.973020 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970921 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-node-log\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.973020 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970936 2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 30 00:01:54.973020 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.970953 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b54c3644-ab98-4033-aab9-9cd95993a4c7-tmp\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:54.973020 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.971044 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-cni-binary-copy\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:54.973020 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.971082 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng6wn\" (UniqueName: \"kubernetes.io/projected/5bfc194e-138e-4335-87f4-f6837c8f2e49-kube-api-access-ng6wn\") pod \"multus-additional-cni-plugins-lg8hw\" (UID: \"5bfc194e-138e-4335-87f4-f6837c8f2e49\") " pod="openshift-multus/multus-additional-cni-plugins-lg8hw" Jul 30 00:01:54.973020 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.971110 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-sys\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:54.973020 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.971154 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b54c3644-ab98-4033-aab9-9cd95993a4c7-etc-tuned\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:54.973020 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.971180 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-host-var-lib-kubelet\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:54.973020 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.971207 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f48f6e0-a27a-413d-9033-19a19f9a5331-socket-dir\") pod \"aws-ebs-csi-driver-node-xxsrg\" (UID: \"4f48f6e0-a27a-413d-9033-19a19f9a5331\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg" Jul 30 00:01:54.973020 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.971235 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f48f6e0-a27a-413d-9033-19a19f9a5331-registration-dir\") pod \"aws-ebs-csi-driver-node-xxsrg\" (UID: \"4f48f6e0-a27a-413d-9033-19a19f9a5331\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg" Jul 30 00:01:54.973020 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.971261 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-systemd-units\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.973020 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.971356 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-systemd-units\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.973020 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.971612 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4f48f6e0-a27a-413d-9033-19a19f9a5331-registration-dir\") pod \"aws-ebs-csi-driver-node-xxsrg\" (UID: \"4f48f6e0-a27a-413d-9033-19a19f9a5331\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg" Jul 30 00:01:54.973020 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.971724 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4f48f6e0-a27a-413d-9033-19a19f9a5331-socket-dir\") pod \"aws-ebs-csi-driver-node-xxsrg\" (UID: \"4f48f6e0-a27a-413d-9033-19a19f9a5331\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg" Jul 30 00:01:54.975472 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.975448 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-ovn-node-metrics-cert\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.980641 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:54.980619 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 00:01:54.980641 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:54.980645 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 00:01:54.980824 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:54.980659 2572 projected.go:196] Error preparing data for projected volume kube-api-access-m5mhv for pod openshift-network-diagnostics/network-check-target-xpx87: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 00:01:54.980824 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:54.980728 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/375724f6-f976-4dc0-a53a-a50fca8d270c-kube-api-access-m5mhv podName:375724f6-f976-4dc0-a53a-a50fca8d270c nodeName:}" failed. No retries permitted until 2026-07-30 00:01:55.480707967 +0000 UTC m=+2.983966224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m5mhv" (UniqueName: "kubernetes.io/projected/375724f6-f976-4dc0-a53a-a50fca8d270c-kube-api-access-m5mhv") pod "network-check-target-xpx87" (UID: "375724f6-f976-4dc0-a53a-a50fca8d270c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 00:01:54.983633 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.983557 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqgzj\" (UniqueName: \"kubernetes.io/projected/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-kube-api-access-cqgzj\") pod \"network-metrics-daemon-nk989\" (UID: \"a38e0c1c-7133-40ff-9ba4-9a066da5c41e\") " pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:01:54.984098 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.983985 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5crh\" (UniqueName: \"kubernetes.io/projected/aa41535c-676c-430a-9151-7257622c4076-kube-api-access-l5crh\") pod \"node-ca-lbgfr\" (UID: \"aa41535c-676c-430a-9151-7257622c4076\") " pod="openshift-image-registry/node-ca-lbgfr" Jul 30 00:01:54.984098 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.983983 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bltj\" (UniqueName: \"kubernetes.io/projected/4f48f6e0-a27a-413d-9033-19a19f9a5331-kube-api-access-8bltj\") pod \"aws-ebs-csi-driver-node-xxsrg\" (UID: \"4f48f6e0-a27a-413d-9033-19a19f9a5331\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg" Jul 30 00:01:54.984098 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.984032 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrgck\" (UniqueName: \"kubernetes.io/projected/29dfb1a0-5edf-449a-aaaa-5f46f319fba1-kube-api-access-rrgck\") pod \"ovnkube-node-l48hb\" (UID: \"29dfb1a0-5edf-449a-aaaa-5f46f319fba1\") " pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:54.984098 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.984053 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng6wn\" (UniqueName: \"kubernetes.io/projected/5bfc194e-138e-4335-87f4-f6837c8f2e49-kube-api-access-ng6wn\") pod \"multus-additional-cni-plugins-lg8hw\" (UID: \"5bfc194e-138e-4335-87f4-f6837c8f2e49\") " pod="openshift-multus/multus-additional-cni-plugins-lg8hw" Jul 30 00:01:54.984360 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.984153 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rd44\" (UniqueName: \"kubernetes.io/projected/6bf193c2-d465-4e6c-abaf-afc0f08815fa-kube-api-access-2rd44\") pod \"node-resolver-zsmpb\" (UID: \"6bf193c2-d465-4e6c-abaf-afc0f08815fa\") " pod="openshift-dns/node-resolver-zsmpb" Jul 30 00:01:54.992627 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.992571 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-118.ec2.internal" event={"ID":"7ffdecdbcec9362ce0d8e8d2332c8e14","Type":"ContainerStarted","Data":"c3766633f62b6e5297cff344314d22e4149337b53e839bb23129f9dbe7f967f0"} Jul 30 00:01:54.994437 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:54.994395 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-118.ec2.internal" event={"ID":"60d5ee2ac122fad92c39ced53363933e","Type":"ContainerStarted","Data":"0b7f848887c7773fce56d39590c29e136f2ae33d3691ac82f9523aa3a4224244"} Jul 30 00:01:55.072123 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.072081 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-etc-sysconfig\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.072333 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.072151 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-etc-kubernetes\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.072333 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.072181 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-run\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.072333 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.072206 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b54c3644-ab98-4033-aab9-9cd95993a4c7-tmp\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.072333 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.072233 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-cni-binary-copy\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.072333 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.072251 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-etc-sysconfig\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.072333 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.072283 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-etc-kubernetes\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.072333 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.072310 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-sys\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.072333 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.072259 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-sys\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.072725 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.072355 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b54c3644-ab98-4033-aab9-9cd95993a4c7-etc-tuned\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.072725 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.072382 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-host-var-lib-kubelet\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.072725 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.072412 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ccd8796b-bcd4-41e9-a946-c43b4c9863cf-host-slash\") pod \"iptables-alerter-v5bld\" (UID: \"ccd8796b-bcd4-41e9-a946-c43b4c9863cf\") " pod="openshift-network-operator/iptables-alerter-v5bld" Jul 30 00:01:55.072725 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.072441 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-etc-sysctl-conf\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.072725 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.072466 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-host\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.072725 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.072449 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-run\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.072725 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.072498 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-host-run-k8s-cni-cncf-io\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.072725 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.072550 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-host-run-k8s-cni-cncf-io\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.072725 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.072559 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46aec834-30bb-4d73-b1b7-f033193e6e99-original-pull-secret\") pod \"global-pull-secret-syncer-77hgn\" (UID: \"46aec834-30bb-4d73-b1b7-f033193e6e99\") " pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:01:55.072725 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.072588 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7nsm\" (UniqueName: \"kubernetes.io/projected/ccd8796b-bcd4-41e9-a946-c43b4c9863cf-kube-api-access-l7nsm\") pod \"iptables-alerter-v5bld\" (UID: \"ccd8796b-bcd4-41e9-a946-c43b4c9863cf\") " pod="openshift-network-operator/iptables-alerter-v5bld" Jul 30 00:01:55.072725 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.072607 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-host-var-lib-cni-bin\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.072725 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.072628 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-host-run-multus-certs\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.072725 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.072651 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f1f125ac-f373-4ad9-8734-263caec79393-agent-certs\") pod \"konnectivity-agent-g7zvw\" (UID: \"f1f125ac-f373-4ad9-8734-263caec79393\") " pod="kube-system/konnectivity-agent-g7zvw" Jul 30 00:01:55.072725 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.072667 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5fgs\" (UniqueName: \"kubernetes.io/projected/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-kube-api-access-f5fgs\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.072725 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.072693 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/46aec834-30bb-4d73-b1b7-f033193e6e99-dbus\") pod \"global-pull-secret-syncer-77hgn\" (UID: \"46aec834-30bb-4d73-b1b7-f033193e6e99\") " pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:01:55.072725 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.072723 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r79z4\" (UniqueName: \"kubernetes.io/projected/b54c3644-ab98-4033-aab9-9cd95993a4c7-kube-api-access-r79z4\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.073507 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.072739 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ccd8796b-bcd4-41e9-a946-c43b4c9863cf-host-slash\") pod \"iptables-alerter-v5bld\" (UID: \"ccd8796b-bcd4-41e9-a946-c43b4c9863cf\") " pod="openshift-network-operator/iptables-alerter-v5bld" Jul 30 00:01:55.073507 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073321 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-host-var-lib-cni-bin\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.073507 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073314 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/46aec834-30bb-4d73-b1b7-f033193e6e99-dbus\") pod \"global-pull-secret-syncer-77hgn\" (UID: \"46aec834-30bb-4d73-b1b7-f033193e6e99\") " pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:01:55.073507 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.072748 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-system-cni-dir\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.073507 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073365 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-host-var-lib-kubelet\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.073507 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073377 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-host-run-netns\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.073507 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:55.073398 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 00:01:55.073507 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073412 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-host-run-netns\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.073507 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073341 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-cni-binary-copy\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.073507 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073423 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-etc-sysctl-d\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.073507 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:55.073465 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46aec834-30bb-4d73-b1b7-f033193e6e99-original-pull-secret podName:46aec834-30bb-4d73-b1b7-f033193e6e99 nodeName:}" failed. No retries permitted until 2026-07-30 00:01:55.573443167 +0000 UTC m=+3.076701412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46aec834-30bb-4d73-b1b7-f033193e6e99-original-pull-secret") pod "global-pull-secret-syncer-77hgn" (UID: "46aec834-30bb-4d73-b1b7-f033193e6e99") : object "kube-system"/"original-pull-secret" not registered Jul 30 00:01:55.073507 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073475 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-host-run-multus-certs\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.073507 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073514 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-etc-systemd\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.074203 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073539 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-cnibin\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.074203 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073565 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-host\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.074203 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073601 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-etc-sysctl-conf\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.074203 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073612 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-etc-systemd\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.074203 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073515 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-etc-sysctl-d\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.074203 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073569 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-lib-modules\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.074203 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073654 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-cnibin\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.074203 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073666 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-lib-modules\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.074203 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073682 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-system-cni-dir\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.074203 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073681 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-var-lib-kubelet\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.074203 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073725 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-var-lib-kubelet\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.074203 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073733 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/46aec834-30bb-4d73-b1b7-f033193e6e99-kubelet-config\") pod \"global-pull-secret-syncer-77hgn\" (UID: \"46aec834-30bb-4d73-b1b7-f033193e6e99\") " pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:01:55.074203 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073761 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-etc-kubernetes\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.074203 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073792 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-etc-modprobe-d\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.074203 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073809 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/46aec834-30bb-4d73-b1b7-f033193e6e99-kubelet-config\") pod \"global-pull-secret-syncer-77hgn\" (UID: \"46aec834-30bb-4d73-b1b7-f033193e6e99\") " pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:01:55.074203 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073819 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-multus-daemon-config\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.074203 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073859 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-etc-kubernetes\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.074203 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073888 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-host-var-lib-cni-multus\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.075084 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073912 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b54c3644-ab98-4033-aab9-9cd95993a4c7-etc-modprobe-d\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.075084 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073935 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f1f125ac-f373-4ad9-8734-263caec79393-konnectivity-ca\") pod \"konnectivity-agent-g7zvw\" (UID: \"f1f125ac-f373-4ad9-8734-263caec79393\") " pod="kube-system/konnectivity-agent-g7zvw" Jul 30 00:01:55.075084 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073961 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-multus-cni-dir\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.075084 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073960 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-host-var-lib-cni-multus\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.075084 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.073985 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-multus-socket-dir-parent\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.075084 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.074052 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-hostroot\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.075084 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.074079 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-multus-conf-dir\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.075084 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.074112 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-multus-cni-dir\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.075084 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.074125 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ccd8796b-bcd4-41e9-a946-c43b4c9863cf-iptables-alerter-script\") pod \"iptables-alerter-v5bld\" (UID: \"ccd8796b-bcd4-41e9-a946-c43b4c9863cf\") " pod="openshift-network-operator/iptables-alerter-v5bld" Jul 30 00:01:55.075084 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.074169 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-os-release\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.075084 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.074198 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-multus-socket-dir-parent\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.075084 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.074260 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-os-release\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.075084 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.074301 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-hostroot\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.075084 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.074333 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-multus-daemon-config\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.075084 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.074393 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-multus-conf-dir\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.075084 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.074620 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f1f125ac-f373-4ad9-8734-263caec79393-konnectivity-ca\") pod \"konnectivity-agent-g7zvw\" (UID: \"f1f125ac-f373-4ad9-8734-263caec79393\") " pod="kube-system/konnectivity-agent-g7zvw" Jul 30 00:01:55.075084 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.074772 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ccd8796b-bcd4-41e9-a946-c43b4c9863cf-iptables-alerter-script\") pod \"iptables-alerter-v5bld\" (UID: \"ccd8796b-bcd4-41e9-a946-c43b4c9863cf\") " pod="openshift-network-operator/iptables-alerter-v5bld" Jul 30 00:01:55.076986 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.076958 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b54c3644-ab98-4033-aab9-9cd95993a4c7-etc-tuned\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.077110 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.077048 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b54c3644-ab98-4033-aab9-9cd95993a4c7-tmp\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.078183 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.078147 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f1f125ac-f373-4ad9-8734-263caec79393-agent-certs\") pod \"konnectivity-agent-g7zvw\" (UID: \"f1f125ac-f373-4ad9-8734-263caec79393\") " pod="kube-system/konnectivity-agent-g7zvw" Jul 30 00:01:55.085375 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.085309 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7nsm\" (UniqueName: \"kubernetes.io/projected/ccd8796b-bcd4-41e9-a946-c43b4c9863cf-kube-api-access-l7nsm\") pod \"iptables-alerter-v5bld\" (UID: \"ccd8796b-bcd4-41e9-a946-c43b4c9863cf\") " pod="openshift-network-operator/iptables-alerter-v5bld" Jul 30 00:01:55.086238 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.086210 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5fgs\" (UniqueName: \"kubernetes.io/projected/afff0c7f-4d59-4a4f-bc4e-2dcdb2521463-kube-api-access-f5fgs\") pod \"multus-r2jkk\" (UID: \"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463\") " pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.088570 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.088211 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r79z4\" (UniqueName: \"kubernetes.io/projected/b54c3644-ab98-4033-aab9-9cd95993a4c7-kube-api-access-r79z4\") pod \"tuned-652c8\" (UID: \"b54c3644-ab98-4033-aab9-9cd95993a4c7\") " pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.162748 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.162706 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg" Jul 30 00:01:55.171921 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.171492 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zsmpb" Jul 30 00:01:55.182938 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.182724 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lbgfr" Jul 30 00:01:55.190412 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.190382 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lg8hw" Jul 30 00:01:55.197049 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.197025 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:01:55.203684 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.203649 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g7zvw" Jul 30 00:01:55.211780 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.211751 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-652c8" Jul 30 00:01:55.219941 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.219914 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v5bld" Jul 30 00:01:55.226118 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.226094 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r2jkk" Jul 30 00:01:55.270308 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.270274 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 00:01:55.478190 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.478099 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs\") pod \"network-metrics-daemon-nk989\" (UID: \"a38e0c1c-7133-40ff-9ba4-9a066da5c41e\") " pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:01:55.478364 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:55.478290 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 00:01:55.478442 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:55.478379 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs podName:a38e0c1c-7133-40ff-9ba4-9a066da5c41e nodeName:}" failed. No retries permitted until 2026-07-30 00:01:56.478355438 +0000 UTC m=+3.981613682 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs") pod "network-metrics-daemon-nk989" (UID: "a38e0c1c-7133-40ff-9ba4-9a066da5c41e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 00:01:55.578747 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.578711 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5mhv\" (UniqueName: \"kubernetes.io/projected/375724f6-f976-4dc0-a53a-a50fca8d270c-kube-api-access-m5mhv\") pod \"network-check-target-xpx87\" (UID: \"375724f6-f976-4dc0-a53a-a50fca8d270c\") " pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:01:55.578881 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.578800 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46aec834-30bb-4d73-b1b7-f033193e6e99-original-pull-secret\") pod \"global-pull-secret-syncer-77hgn\" (UID: \"46aec834-30bb-4d73-b1b7-f033193e6e99\") " pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:01:55.578934 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:55.578893 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 00:01:55.578934 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:55.578915 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 00:01:55.578934 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:55.578925 2572 projected.go:196] Error preparing data for projected volume kube-api-access-m5mhv for pod openshift-network-diagnostics/network-check-target-xpx87: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 00:01:55.579051 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:55.578944 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 00:01:55.579051 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:55.578985 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/375724f6-f976-4dc0-a53a-a50fca8d270c-kube-api-access-m5mhv podName:375724f6-f976-4dc0-a53a-a50fca8d270c nodeName:}" failed. No retries permitted until 2026-07-30 00:01:56.57896915 +0000 UTC m=+4.082227389 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m5mhv" (UniqueName: "kubernetes.io/projected/375724f6-f976-4dc0-a53a-a50fca8d270c-kube-api-access-m5mhv") pod "network-check-target-xpx87" (UID: "375724f6-f976-4dc0-a53a-a50fca8d270c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 00:01:55.579051 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:55.579010 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46aec834-30bb-4d73-b1b7-f033193e6e99-original-pull-secret podName:46aec834-30bb-4d73-b1b7-f033193e6e99 nodeName:}" failed. No retries permitted until 2026-07-30 00:01:56.578996232 +0000 UTC m=+4.082254471 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46aec834-30bb-4d73-b1b7-f033193e6e99-original-pull-secret") pod "global-pull-secret-syncer-77hgn" (UID: "46aec834-30bb-4d73-b1b7-f033193e6e99") : object "kube-system"/"original-pull-secret" not registered Jul 30 00:01:55.900845 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.900754 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 23:56:55 +0000 UTC" deadline="2028-04-01 08:12:31.673774358 +0000 UTC" Jul 30 00:01:55.900845 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.900795 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14672h10m35.772983765s" Jul 30 00:01:55.913394 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:55.913357 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29dfb1a0_5edf_449a_aaaa_5f46f319fba1.slice/crio-465736231d60d0ce01e20ebf3b8d7d3e495a230389e70ad1f6daca91b2c006f5 WatchSource:0}: Error finding container 465736231d60d0ce01e20ebf3b8d7d3e495a230389e70ad1f6daca91b2c006f5: Status 404 returned error can't find the container with id 465736231d60d0ce01e20ebf3b8d7d3e495a230389e70ad1f6daca91b2c006f5 Jul 30 00:01:55.914425 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:55.914397 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bf193c2_d465_4e6c_abaf_afc0f08815fa.slice/crio-6d5babf9d872fb726a371861e902e6f7e50b3ebf7a928e48f5718992c94861a6 WatchSource:0}: Error finding container 6d5babf9d872fb726a371861e902e6f7e50b3ebf7a928e48f5718992c94861a6: Status 404 returned error can't find the container with id 6d5babf9d872fb726a371861e902e6f7e50b3ebf7a928e48f5718992c94861a6 Jul 30 00:01:55.917795 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:55.917775 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f125ac_f373_4ad9_8734_263caec79393.slice/crio-9bf5b762f2e07d4b49998a73459d1751aa45a47b804e9971b63c7c08ec0b64cb WatchSource:0}: Error finding container 9bf5b762f2e07d4b49998a73459d1751aa45a47b804e9971b63c7c08ec0b64cb: Status 404 returned error can't find the container with id 9bf5b762f2e07d4b49998a73459d1751aa45a47b804e9971b63c7c08ec0b64cb Jul 30 00:01:55.987585 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.987545 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:01:55.987760 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:55.987685 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77hgn" podUID="46aec834-30bb-4d73-b1b7-f033193e6e99" Jul 30 00:01:55.997384 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.997335 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g7zvw" event={"ID":"f1f125ac-f373-4ad9-8734-263caec79393","Type":"ContainerStarted","Data":"9bf5b762f2e07d4b49998a73459d1751aa45a47b804e9971b63c7c08ec0b64cb"} Jul 30 00:01:55.998609 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.998582 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zsmpb" event={"ID":"6bf193c2-d465-4e6c-abaf-afc0f08815fa","Type":"ContainerStarted","Data":"6d5babf9d872fb726a371861e902e6f7e50b3ebf7a928e48f5718992c94861a6"} Jul 30 00:01:55.999767 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:55.999736 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" event={"ID":"29dfb1a0-5edf-449a-aaaa-5f46f319fba1","Type":"ContainerStarted","Data":"465736231d60d0ce01e20ebf3b8d7d3e495a230389e70ad1f6daca91b2c006f5"} Jul 30 00:01:56.260486 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:56.260454 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd8796b_bcd4_41e9_a946_c43b4c9863cf.slice/crio-e0e5a39470eedf639c362e840fe8b19ab9022f8961fe5cb6b87ff9551670570a WatchSource:0}: Error finding container e0e5a39470eedf639c362e840fe8b19ab9022f8961fe5cb6b87ff9551670570a: Status 404 returned error can't find the container with id e0e5a39470eedf639c362e840fe8b19ab9022f8961fe5cb6b87ff9551670570a Jul 30 00:01:56.262057 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:56.262023 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb54c3644_ab98_4033_aab9_9cd95993a4c7.slice/crio-c6f1f80466a738228b4717c8ca9d1e2ec724efd157eece5a4fe39c608192ef0b WatchSource:0}: Error finding container c6f1f80466a738228b4717c8ca9d1e2ec724efd157eece5a4fe39c608192ef0b: Status 404 returned error can't find the container with id c6f1f80466a738228b4717c8ca9d1e2ec724efd157eece5a4fe39c608192ef0b Jul 30 00:01:56.263580 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:56.263208 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bfc194e_138e_4335_87f4_f6837c8f2e49.slice/crio-881de3963ac870e547d7a9bd4ced4bed34309f190d73c3ece9b89c87b3f7d4c6 WatchSource:0}: Error finding container 881de3963ac870e547d7a9bd4ced4bed34309f190d73c3ece9b89c87b3f7d4c6: Status 404 returned error can't find the container with id 881de3963ac870e547d7a9bd4ced4bed34309f190d73c3ece9b89c87b3f7d4c6 Jul 30 00:01:56.263580 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:56.263489 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa41535c_676c_430a_9151_7257622c4076.slice/crio-9d7303f905d3b8213d6e830c404976590b8351799f8d673124f49ce7c345471b WatchSource:0}: Error finding container 9d7303f905d3b8213d6e830c404976590b8351799f8d673124f49ce7c345471b: Status 404 returned error can't find the container with id 9d7303f905d3b8213d6e830c404976590b8351799f8d673124f49ce7c345471b Jul 30 00:01:56.265309 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:56.265166 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafff0c7f_4d59_4a4f_bc4e_2dcdb2521463.slice/crio-28b3323c6be1e669d82866ac4b345da90f0036044971e80996641947075963e5 WatchSource:0}: Error finding container 28b3323c6be1e669d82866ac4b345da90f0036044971e80996641947075963e5: Status 404 returned error can't find the container with id 28b3323c6be1e669d82866ac4b345da90f0036044971e80996641947075963e5 Jul 30 00:01:56.266745 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:01:56.266720 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f48f6e0_a27a_413d_9033_19a19f9a5331.slice/crio-c040dd382b70c1f1bb5154e16e30c8bc0cd0054d424d589ba8cd6a8d04797b6f WatchSource:0}: Error finding container c040dd382b70c1f1bb5154e16e30c8bc0cd0054d424d589ba8cd6a8d04797b6f: Status 404 returned error can't find the container with id c040dd382b70c1f1bb5154e16e30c8bc0cd0054d424d589ba8cd6a8d04797b6f Jul 30 00:01:56.486314 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:56.486025 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs\") pod \"network-metrics-daemon-nk989\" (UID: \"a38e0c1c-7133-40ff-9ba4-9a066da5c41e\") " pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:01:56.486314 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:56.486217 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 00:01:56.486314 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:56.486306 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs podName:a38e0c1c-7133-40ff-9ba4-9a066da5c41e nodeName:}" failed. No retries permitted until 2026-07-30 00:01:58.486282146 +0000 UTC m=+5.989540400 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs") pod "network-metrics-daemon-nk989" (UID: "a38e0c1c-7133-40ff-9ba4-9a066da5c41e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 00:01:56.587750 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:56.587695 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5mhv\" (UniqueName: \"kubernetes.io/projected/375724f6-f976-4dc0-a53a-a50fca8d270c-kube-api-access-m5mhv\") pod \"network-check-target-xpx87\" (UID: \"375724f6-f976-4dc0-a53a-a50fca8d270c\") " pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:01:56.587968 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:56.587781 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46aec834-30bb-4d73-b1b7-f033193e6e99-original-pull-secret\") pod \"global-pull-secret-syncer-77hgn\" (UID: \"46aec834-30bb-4d73-b1b7-f033193e6e99\") " pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:01:56.587968 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:56.587913 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 00:01:56.587968 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:56.587955 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 00:01:56.588152 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:56.587972 2572 projected.go:196] Error preparing data for projected volume kube-api-access-m5mhv for pod openshift-network-diagnostics/network-check-target-xpx87: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 00:01:56.588152 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:56.587917 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 00:01:56.588152 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:56.588055 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/375724f6-f976-4dc0-a53a-a50fca8d270c-kube-api-access-m5mhv podName:375724f6-f976-4dc0-a53a-a50fca8d270c nodeName:}" failed. No retries permitted until 2026-07-30 00:01:58.588029008 +0000 UTC m=+6.091287272 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-m5mhv" (UniqueName: "kubernetes.io/projected/375724f6-f976-4dc0-a53a-a50fca8d270c-kube-api-access-m5mhv") pod "network-check-target-xpx87" (UID: "375724f6-f976-4dc0-a53a-a50fca8d270c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 00:01:56.588152 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:56.588085 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46aec834-30bb-4d73-b1b7-f033193e6e99-original-pull-secret podName:46aec834-30bb-4d73-b1b7-f033193e6e99 nodeName:}" failed. No retries permitted until 2026-07-30 00:01:58.588071589 +0000 UTC m=+6.091329829 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46aec834-30bb-4d73-b1b7-f033193e6e99-original-pull-secret") pod "global-pull-secret-syncer-77hgn" (UID: "46aec834-30bb-4d73-b1b7-f033193e6e99") : object "kube-system"/"original-pull-secret" not registered Jul 30 00:01:56.902046 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:56.901909 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 23:56:55 +0000 UTC" deadline="2027-12-31 03:50:48.66416638 +0000 UTC" Jul 30 00:01:56.902046 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:56.901987 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12459h48m51.76218386s" Jul 30 00:01:56.989927 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:56.989886 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:01:56.990134 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:56.990043 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpx87" podUID="375724f6-f976-4dc0-a53a-a50fca8d270c" Jul 30 00:01:56.990370 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:56.990353 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:01:56.990490 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:56.990466 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nk989" podUID="a38e0c1c-7133-40ff-9ba4-9a066da5c41e" Jul 30 00:01:57.004088 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:57.004044 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-652c8" event={"ID":"b54c3644-ab98-4033-aab9-9cd95993a4c7","Type":"ContainerStarted","Data":"c6f1f80466a738228b4717c8ca9d1e2ec724efd157eece5a4fe39c608192ef0b"} Jul 30 00:01:57.005797 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:57.005762 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v5bld" event={"ID":"ccd8796b-bcd4-41e9-a946-c43b4c9863cf","Type":"ContainerStarted","Data":"e0e5a39470eedf639c362e840fe8b19ab9022f8961fe5cb6b87ff9551670570a"} Jul 30 00:01:57.012493 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:57.012430 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-118.ec2.internal" event={"ID":"60d5ee2ac122fad92c39ced53363933e","Type":"ContainerStarted","Data":"11c343836d9902e07a76fc536a10f9193f13b67015d54b0352d0497e7789922c"} Jul 30 00:01:57.017262 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:57.016530 2572 generic.go:356] "Generic (PLEG): container finished" podID="7ffdecdbcec9362ce0d8e8d2332c8e14" containerID="4913ab608dc09639fdc8d34515655fdb15927a6b9a7ce4ef0f0fcd3d307c521c" exitCode=0 Jul 30 00:01:57.017262 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:57.016656 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-118.ec2.internal" event={"ID":"7ffdecdbcec9362ce0d8e8d2332c8e14","Type":"ContainerDied","Data":"4913ab608dc09639fdc8d34515655fdb15927a6b9a7ce4ef0f0fcd3d307c521c"} Jul 30 00:01:57.018825 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:57.018788 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lg8hw" event={"ID":"5bfc194e-138e-4335-87f4-f6837c8f2e49","Type":"ContainerStarted","Data":"881de3963ac870e547d7a9bd4ced4bed34309f190d73c3ece9b89c87b3f7d4c6"} Jul 30 00:01:57.021672 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:57.021598 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg" event={"ID":"4f48f6e0-a27a-413d-9033-19a19f9a5331","Type":"ContainerStarted","Data":"c040dd382b70c1f1bb5154e16e30c8bc0cd0054d424d589ba8cd6a8d04797b6f"} Jul 30 00:01:57.024088 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:57.024057 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lbgfr" event={"ID":"aa41535c-676c-430a-9151-7257622c4076","Type":"ContainerStarted","Data":"9d7303f905d3b8213d6e830c404976590b8351799f8d673124f49ce7c345471b"} Jul 30 00:01:57.026839 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:57.026759 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r2jkk" event={"ID":"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463","Type":"ContainerStarted","Data":"28b3323c6be1e669d82866ac4b345da90f0036044971e80996641947075963e5"} Jul 30 00:01:57.047174 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:57.047077 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-118.ec2.internal" podStartSLOduration=3.04705229 podStartE2EDuration="3.04705229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:01:57.029715174 +0000 UTC m=+4.532973438" watchObservedRunningTime="2026-07-30 00:01:57.04705229 +0000 UTC m=+4.550310555" Jul 30 00:01:57.988008 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:57.987411 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:01:57.988008 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:57.987563 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77hgn" podUID="46aec834-30bb-4d73-b1b7-f033193e6e99" Jul 30 00:01:58.046232 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:58.046165 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-118.ec2.internal" event={"ID":"7ffdecdbcec9362ce0d8e8d2332c8e14","Type":"ContainerStarted","Data":"6f7caadcc63c77fdbe54b99363cd4d8cb60b8793445ad172e4b01d71aac65fa1"} Jul 30 00:01:58.062252 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:58.061436 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-118.ec2.internal" podStartSLOduration=4.061411395 podStartE2EDuration="4.061411395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:01:58.060786839 +0000 UTC m=+5.564045105" watchObservedRunningTime="2026-07-30 00:01:58.061411395 +0000 UTC m=+5.564669659" Jul 30 00:01:58.505021 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:58.504982 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs\") pod \"network-metrics-daemon-nk989\" (UID: \"a38e0c1c-7133-40ff-9ba4-9a066da5c41e\") " pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:01:58.505277 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:58.505257 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 00:01:58.505361 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:58.505335 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs podName:a38e0c1c-7133-40ff-9ba4-9a066da5c41e nodeName:}" failed. No retries permitted until 2026-07-30 00:02:02.505312963 +0000 UTC m=+10.008571244 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs") pod "network-metrics-daemon-nk989" (UID: "a38e0c1c-7133-40ff-9ba4-9a066da5c41e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 00:01:58.609074 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:58.608181 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5mhv\" (UniqueName: \"kubernetes.io/projected/375724f6-f976-4dc0-a53a-a50fca8d270c-kube-api-access-m5mhv\") pod \"network-check-target-xpx87\" (UID: \"375724f6-f976-4dc0-a53a-a50fca8d270c\") " pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:01:58.609074 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:58.608254 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46aec834-30bb-4d73-b1b7-f033193e6e99-original-pull-secret\") pod \"global-pull-secret-syncer-77hgn\" (UID: \"46aec834-30bb-4d73-b1b7-f033193e6e99\") " pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:01:58.609074 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:58.608402 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 00:01:58.609074 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:58.608466 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46aec834-30bb-4d73-b1b7-f033193e6e99-original-pull-secret podName:46aec834-30bb-4d73-b1b7-f033193e6e99 nodeName:}" failed. No retries permitted until 2026-07-30 00:02:02.608447325 +0000 UTC m=+10.111705569 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46aec834-30bb-4d73-b1b7-f033193e6e99-original-pull-secret") pod "global-pull-secret-syncer-77hgn" (UID: "46aec834-30bb-4d73-b1b7-f033193e6e99") : object "kube-system"/"original-pull-secret" not registered Jul 30 00:01:58.609074 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:58.608897 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 00:01:58.609074 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:58.608917 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 00:01:58.609074 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:58.608932 2572 projected.go:196] Error preparing data for projected volume kube-api-access-m5mhv for pod openshift-network-diagnostics/network-check-target-xpx87: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 00:01:58.609074 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:58.608978 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/375724f6-f976-4dc0-a53a-a50fca8d270c-kube-api-access-m5mhv podName:375724f6-f976-4dc0-a53a-a50fca8d270c nodeName:}" failed. No retries permitted until 2026-07-30 00:02:02.608963342 +0000 UTC m=+10.112221585 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-m5mhv" (UniqueName: "kubernetes.io/projected/375724f6-f976-4dc0-a53a-a50fca8d270c-kube-api-access-m5mhv") pod "network-check-target-xpx87" (UID: "375724f6-f976-4dc0-a53a-a50fca8d270c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 00:01:58.987243 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:58.986944 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:01:58.987243 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:58.987106 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nk989" podUID="a38e0c1c-7133-40ff-9ba4-9a066da5c41e" Jul 30 00:01:58.988086 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:58.987870 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:01:58.988086 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:58.988005 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpx87" podUID="375724f6-f976-4dc0-a53a-a50fca8d270c" Jul 30 00:01:59.988423 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:01:59.988364 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:01:59.988918 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:01:59.988607 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77hgn" podUID="46aec834-30bb-4d73-b1b7-f033193e6e99" Jul 30 00:02:00.986981 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:00.986948 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:02:00.987217 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:00.987082 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpx87" podUID="375724f6-f976-4dc0-a53a-a50fca8d270c" Jul 30 00:02:00.987297 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:00.987252 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:02:00.987416 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:00.987381 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nk989" podUID="a38e0c1c-7133-40ff-9ba4-9a066da5c41e" Jul 30 00:02:01.986834 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:01.986754 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:02:01.987343 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:01.986902 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77hgn" podUID="46aec834-30bb-4d73-b1b7-f033193e6e99" Jul 30 00:02:02.549461 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:02.549396 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs\") pod \"network-metrics-daemon-nk989\" (UID: \"a38e0c1c-7133-40ff-9ba4-9a066da5c41e\") " pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:02:02.549649 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:02.549559 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 00:02:02.549649 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:02.549637 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs podName:a38e0c1c-7133-40ff-9ba4-9a066da5c41e nodeName:}" failed. No retries permitted until 2026-07-30 00:02:10.549618534 +0000 UTC m=+18.052876775 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs") pod "network-metrics-daemon-nk989" (UID: "a38e0c1c-7133-40ff-9ba4-9a066da5c41e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 00:02:02.650675 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:02.650640 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46aec834-30bb-4d73-b1b7-f033193e6e99-original-pull-secret\") pod \"global-pull-secret-syncer-77hgn\" (UID: \"46aec834-30bb-4d73-b1b7-f033193e6e99\") " pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:02:02.650859 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:02.650733 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5mhv\" (UniqueName: \"kubernetes.io/projected/375724f6-f976-4dc0-a53a-a50fca8d270c-kube-api-access-m5mhv\") pod \"network-check-target-xpx87\" (UID: \"375724f6-f976-4dc0-a53a-a50fca8d270c\") " pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:02:02.650859 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:02.650815 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 00:02:02.650988 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:02.650902 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46aec834-30bb-4d73-b1b7-f033193e6e99-original-pull-secret podName:46aec834-30bb-4d73-b1b7-f033193e6e99 nodeName:}" failed. No retries permitted until 2026-07-30 00:02:10.650880865 +0000 UTC m=+18.154139105 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46aec834-30bb-4d73-b1b7-f033193e6e99-original-pull-secret") pod "global-pull-secret-syncer-77hgn" (UID: "46aec834-30bb-4d73-b1b7-f033193e6e99") : object "kube-system"/"original-pull-secret" not registered Jul 30 00:02:02.650988 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:02.650907 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 00:02:02.650988 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:02.650929 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 00:02:02.650988 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:02.650943 2572 projected.go:196] Error preparing data for projected volume kube-api-access-m5mhv for pod openshift-network-diagnostics/network-check-target-xpx87: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 00:02:02.651288 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:02.651000 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/375724f6-f976-4dc0-a53a-a50fca8d270c-kube-api-access-m5mhv podName:375724f6-f976-4dc0-a53a-a50fca8d270c nodeName:}" failed. No retries permitted until 2026-07-30 00:02:10.650984069 +0000 UTC m=+18.154242324 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-m5mhv" (UniqueName: "kubernetes.io/projected/375724f6-f976-4dc0-a53a-a50fca8d270c-kube-api-access-m5mhv") pod "network-check-target-xpx87" (UID: "375724f6-f976-4dc0-a53a-a50fca8d270c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 00:02:02.988746 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:02.988379 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:02:02.988746 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:02.988512 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nk989" podUID="a38e0c1c-7133-40ff-9ba4-9a066da5c41e" Jul 30 00:02:02.989245 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:02.988890 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:02:02.989245 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:02.988967 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpx87" podUID="375724f6-f976-4dc0-a53a-a50fca8d270c" Jul 30 00:02:03.987363 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:03.987326 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:02:03.987566 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:03.987479 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77hgn" podUID="46aec834-30bb-4d73-b1b7-f033193e6e99" Jul 30 00:02:04.987388 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:04.987352 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:02:04.987868 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:04.987511 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nk989" podUID="a38e0c1c-7133-40ff-9ba4-9a066da5c41e" Jul 30 00:02:04.987868 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:04.987606 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:02:04.987868 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:04.987710 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpx87" podUID="375724f6-f976-4dc0-a53a-a50fca8d270c" Jul 30 00:02:05.986677 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:05.986643 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:02:05.986851 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:05.986777 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77hgn" podUID="46aec834-30bb-4d73-b1b7-f033193e6e99" Jul 30 00:02:06.987093 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:06.987005 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:02:06.987547 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:06.987005 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:02:06.987547 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:06.987165 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nk989" podUID="a38e0c1c-7133-40ff-9ba4-9a066da5c41e" Jul 30 00:02:06.987547 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:06.987230 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpx87" podUID="375724f6-f976-4dc0-a53a-a50fca8d270c" Jul 30 00:02:07.987008 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:07.986968 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:02:07.987251 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:07.987087 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77hgn" podUID="46aec834-30bb-4d73-b1b7-f033193e6e99" Jul 30 00:02:08.987502 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:08.987463 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:02:08.987965 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:08.987915 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpx87" podUID="375724f6-f976-4dc0-a53a-a50fca8d270c" Jul 30 00:02:08.988025 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:08.988004 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:02:08.988212 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:08.988192 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nk989" podUID="a38e0c1c-7133-40ff-9ba4-9a066da5c41e" Jul 30 00:02:09.987255 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:09.987219 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:02:09.987490 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:09.987345 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77hgn" podUID="46aec834-30bb-4d73-b1b7-f033193e6e99" Jul 30 00:02:10.614570 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:10.614487 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs\") pod \"network-metrics-daemon-nk989\" (UID: \"a38e0c1c-7133-40ff-9ba4-9a066da5c41e\") " pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:02:10.615015 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:10.614631 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 00:02:10.615015 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:10.614714 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs podName:a38e0c1c-7133-40ff-9ba4-9a066da5c41e nodeName:}" failed. No retries permitted until 2026-07-30 00:02:26.614693466 +0000 UTC m=+34.117951733 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs") pod "network-metrics-daemon-nk989" (UID: "a38e0c1c-7133-40ff-9ba4-9a066da5c41e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 00:02:10.715609 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:10.715574 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46aec834-30bb-4d73-b1b7-f033193e6e99-original-pull-secret\") pod \"global-pull-secret-syncer-77hgn\" (UID: \"46aec834-30bb-4d73-b1b7-f033193e6e99\") " pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:02:10.715803 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:10.715648 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5mhv\" (UniqueName: \"kubernetes.io/projected/375724f6-f976-4dc0-a53a-a50fca8d270c-kube-api-access-m5mhv\") pod \"network-check-target-xpx87\" (UID: \"375724f6-f976-4dc0-a53a-a50fca8d270c\") " pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:02:10.715803 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:10.715755 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 00:02:10.715803 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:10.715765 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 00:02:10.715803 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:10.715789 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 00:02:10.715803 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:10.715803 2572 projected.go:196] Error preparing data for projected volume kube-api-access-m5mhv for pod openshift-network-diagnostics/network-check-target-xpx87: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 00:02:10.716013 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:10.715826 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46aec834-30bb-4d73-b1b7-f033193e6e99-original-pull-secret podName:46aec834-30bb-4d73-b1b7-f033193e6e99 nodeName:}" failed. No retries permitted until 2026-07-30 00:02:26.715809014 +0000 UTC m=+34.219067258 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46aec834-30bb-4d73-b1b7-f033193e6e99-original-pull-secret") pod "global-pull-secret-syncer-77hgn" (UID: "46aec834-30bb-4d73-b1b7-f033193e6e99") : object "kube-system"/"original-pull-secret" not registered Jul 30 00:02:10.716013 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:10.715852 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/375724f6-f976-4dc0-a53a-a50fca8d270c-kube-api-access-m5mhv podName:375724f6-f976-4dc0-a53a-a50fca8d270c nodeName:}" failed. No retries permitted until 2026-07-30 00:02:26.715837474 +0000 UTC m=+34.219095732 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-m5mhv" (UniqueName: "kubernetes.io/projected/375724f6-f976-4dc0-a53a-a50fca8d270c-kube-api-access-m5mhv") pod "network-check-target-xpx87" (UID: "375724f6-f976-4dc0-a53a-a50fca8d270c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 00:02:10.987659 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:10.987568 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:02:10.987659 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:10.987620 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:02:10.987885 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:10.987725 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nk989" podUID="a38e0c1c-7133-40ff-9ba4-9a066da5c41e" Jul 30 00:02:10.987885 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:10.987862 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpx87" podUID="375724f6-f976-4dc0-a53a-a50fca8d270c" Jul 30 00:02:11.987404 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:11.987357 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:02:11.987863 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:11.987486 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77hgn" podUID="46aec834-30bb-4d73-b1b7-f033193e6e99" Jul 30 00:02:12.988596 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:12.988546 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:02:12.989101 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:12.988664 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpx87" podUID="375724f6-f976-4dc0-a53a-a50fca8d270c" Jul 30 00:02:12.989101 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:12.988723 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:02:12.989101 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:12.988847 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nk989" podUID="a38e0c1c-7133-40ff-9ba4-9a066da5c41e" Jul 30 00:02:13.987124 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:13.987069 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:02:13.987336 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:13.987220 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77hgn" podUID="46aec834-30bb-4d73-b1b7-f033193e6e99" Jul 30 00:02:14.987465 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:14.987066 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:02:14.988388 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:14.987078 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:02:14.988388 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:14.987626 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nk989" podUID="a38e0c1c-7133-40ff-9ba4-9a066da5c41e" Jul 30 00:02:14.988388 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:14.987685 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpx87" podUID="375724f6-f976-4dc0-a53a-a50fca8d270c" Jul 30 00:02:15.089263 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:15.089231 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g7zvw" event={"ID":"f1f125ac-f373-4ad9-8734-263caec79393","Type":"ContainerStarted","Data":"df58e8d5e26ccf9f43189ee124fe2ee45e9290ec6bcef9611490826fa53e23b0"} Jul 30 00:02:15.090714 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:15.090681 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-652c8" event={"ID":"b54c3644-ab98-4033-aab9-9cd95993a4c7","Type":"ContainerStarted","Data":"7af7ffcb09bb345d48431d0bf9d1ace139185e48468e28e608276b99d2bbedef"} Jul 30 00:02:15.092043 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:15.092018 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zsmpb" event={"ID":"6bf193c2-d465-4e6c-abaf-afc0f08815fa","Type":"ContainerStarted","Data":"34376197cdf82ff4fce6719865a99454cbccf4125ba461962c8b6a14ba986725"} Jul 30 00:02:15.094430 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:15.094407 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" event={"ID":"29dfb1a0-5edf-449a-aaaa-5f46f319fba1","Type":"ContainerStarted","Data":"839d905cbc47ade2ca4279a074840cb95e987f5b7a5c261fd75b054f492a3971"} Jul 30 00:02:15.094525 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:15.094467 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" event={"ID":"29dfb1a0-5edf-449a-aaaa-5f46f319fba1","Type":"ContainerStarted","Data":"ba87ee438629ab9e066737ff383c0b021172dd9af9b507f747e81e786d47a654"} Jul 30 00:02:15.094525 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:15.094511 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" event={"ID":"29dfb1a0-5edf-449a-aaaa-5f46f319fba1","Type":"ContainerStarted","Data":"9c94f32cfea83172da9f23907e4d0b89a1faa2b30075ee1811254f459126150f"} Jul 30 00:02:15.095794 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:15.095770 2572 generic.go:356] "Generic (PLEG): container finished" podID="5bfc194e-138e-4335-87f4-f6837c8f2e49" containerID="5d88a923aae0064cd89d641a6562e028b0946ab5d3494b388817119a8c19f9cd" exitCode=0 Jul 30 00:02:15.095902 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:15.095840 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lg8hw" event={"ID":"5bfc194e-138e-4335-87f4-f6837c8f2e49","Type":"ContainerDied","Data":"5d88a923aae0064cd89d641a6562e028b0946ab5d3494b388817119a8c19f9cd"} Jul 30 00:02:15.097374 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:15.097351 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg" event={"ID":"4f48f6e0-a27a-413d-9033-19a19f9a5331","Type":"ContainerStarted","Data":"b70b85a0affbe88c8a77be013cb09e9bb0cb6978c271bb9346d8ea222f97233e"} Jul 30 00:02:15.098900 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:15.098878 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lbgfr" event={"ID":"aa41535c-676c-430a-9151-7257622c4076","Type":"ContainerStarted","Data":"82f42bb2e07b4587ec67fbeba2db9b84f150695cc5ee5babe17dcba822aca63d"} Jul 30 00:02:15.100515 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:15.100419 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r2jkk" event={"ID":"afff0c7f-4d59-4a4f-bc4e-2dcdb2521463","Type":"ContainerStarted","Data":"da517300b8892305b4e4adb6234e27e763f92012d7c07a217c54af2c5d258442"} Jul 30 00:02:15.103387 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:15.103352 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-g7zvw" podStartSLOduration=3.359078419 podStartE2EDuration="22.10334019s" totalImagesPullingTime="18.744261771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:02:15.103142389 +0000 UTC m=+22.606400643" watchObservedRunningTime="2026-07-30 00:02:15.10334019 +0000 UTC m=+22.606598450" Jul 30 00:02:15.116650 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:15.116593 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zsmpb" podStartSLOduration=7.682526364 podStartE2EDuration="22.116571762s" totalImagesPullingTime="14.434045398s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:02:15.116256284 +0000 UTC m=+22.619514543" watchObservedRunningTime="2026-07-30 00:02:15.116571762 +0000 UTC m=+22.619830021" Jul 30 00:02:15.131103 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:15.131050 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-652c8" podStartSLOduration=3.729438943 podStartE2EDuration="22.131032255s" totalImagesPullingTime="18.401593312s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:02:15.130797812 +0000 UTC m=+22.634056073" watchObservedRunningTime="2026-07-30 00:02:15.131032255 +0000 UTC m=+22.634290528" Jul 30 00:02:15.143807 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:15.143764 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lbgfr" podStartSLOduration=3.743174518 podStartE2EDuration="22.143748753s" totalImagesPullingTime="18.400574235s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:02:15.143230946 +0000 UTC m=+22.646489205" watchObservedRunningTime="2026-07-30 00:02:15.143748753 +0000 UTC m=+22.647007055" Jul 30 00:02:15.198150 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:15.198096 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-r2jkk" podStartSLOduration=3.744459127 podStartE2EDuration="22.198081852s" totalImagesPullingTime="18.453622725s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:02:15.197846893 +0000 UTC m=+22.701105154" watchObservedRunningTime="2026-07-30 00:02:15.198081852 +0000 UTC m=+22.701340111" Jul 30 00:02:15.255127 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:15.255085 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-g7zvw" Jul 30 00:02:15.255654 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:15.255638 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-g7zvw" Jul 30 00:02:15.987708 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:15.987625 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:02:15.988333 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:15.987732 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77hgn" podUID="46aec834-30bb-4d73-b1b7-f033193e6e99" Jul 30 00:02:16.104519 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:16.104475 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v5bld" event={"ID":"ccd8796b-bcd4-41e9-a946-c43b4c9863cf","Type":"ContainerStarted","Data":"21bc9d1f673480c9d67d8670797feae201b2dc7e4368ee449f0f41e7b9131327"} Jul 30 00:02:16.107255 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:16.107219 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" event={"ID":"29dfb1a0-5edf-449a-aaaa-5f46f319fba1","Type":"ContainerStarted","Data":"cf60dbeeb53838c65347bc97d44361bac9f1f479a7cc763801119bdb11d2abce"} Jul 30 00:02:16.107374 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:16.107295 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" event={"ID":"29dfb1a0-5edf-449a-aaaa-5f46f319fba1","Type":"ContainerStarted","Data":"d8f3ed8ae6d27ccb60240317ad7cc32086c157efcc6577af9d0e4ce5c2f1872f"} Jul 30 00:02:16.107374 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:16.107325 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" event={"ID":"29dfb1a0-5edf-449a-aaaa-5f46f319fba1","Type":"ContainerStarted","Data":"6ab3aeb87c484a6b6e595f4d8649174ff1e45be39d6a10c2aad8f4571235d926"} Jul 30 00:02:16.112207 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:16.112163 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-g7zvw" Jul 30 00:02:16.112842 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:16.112818 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g7zvw" Jul 30 00:02:16.126460 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:16.126401 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-v5bld" podStartSLOduration=9.038674462 podStartE2EDuration="23.126377719s" totalImagesPullingTime="14.087703257s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:02:16.125478196 +0000 UTC m=+23.628736462" watchObservedRunningTime="2026-07-30 00:02:16.126377719 +0000 UTC m=+23.629635980" Jul 30 00:02:16.228159 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:16.228133 2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 30 00:02:16.924027 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:16.923828 2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-30T00:02:16.228149878Z","UUID":"7d2bce62-cc71-4b9b-995f-0c90def0ba9a","Handler":null,"Name":"","Endpoint":""} Jul 30 00:02:16.926323 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:16.926297 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 30 00:02:16.926488 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:16.926346 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 30 00:02:16.987495 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:16.987462 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:02:16.987681 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:16.987462 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:02:16.987681 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:16.987599 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpx87" podUID="375724f6-f976-4dc0-a53a-a50fca8d270c" Jul 30 00:02:16.987681 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:16.987640 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nk989" podUID="a38e0c1c-7133-40ff-9ba4-9a066da5c41e" Jul 30 00:02:17.111606 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:17.111575 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg" event={"ID":"4f48f6e0-a27a-413d-9033-19a19f9a5331","Type":"ContainerStarted","Data":"2cc9e78222cb6b757c5c29f52eb102e511f72a6c4cb8e3df4a44eee3fbc86f3d"} Jul 30 00:02:17.987797 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:17.987530 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:02:17.988302 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:17.987827 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77hgn" podUID="46aec834-30bb-4d73-b1b7-f033193e6e99" Jul 30 00:02:18.118215 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:18.118177 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" event={"ID":"29dfb1a0-5edf-449a-aaaa-5f46f319fba1","Type":"ContainerStarted","Data":"ede01bc2969c271e452bfb9ee38cd2f3d1cf3c7a829a6775dae3e90ceab2206b"} Jul 30 00:02:18.120389 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:18.120353 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg" event={"ID":"4f48f6e0-a27a-413d-9033-19a19f9a5331","Type":"ContainerStarted","Data":"c9a140bfa1f2d3f1b411ddd8bd9c1553dd7fcee9959c8d64d4ed8ae37c212ef2"} Jul 30 00:02:18.139149 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:18.138052 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xxsrg" podStartSLOduration=4.09329349 podStartE2EDuration="25.138033631s" totalImagesPullingTime="21.044740141s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:01:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:02:18.137601565 +0000 UTC m=+25.640859826" watchObservedRunningTime="2026-07-30 00:02:18.138033631 +0000 UTC m=+25.641291896" Jul 30 00:02:18.987007 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:18.986968 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:02:18.987243 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:18.987101 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nk989" podUID="a38e0c1c-7133-40ff-9ba4-9a066da5c41e" Jul 30 00:02:18.987243 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:18.987137 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:02:18.987347 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:18.987250 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpx87" podUID="375724f6-f976-4dc0-a53a-a50fca8d270c" Jul 30 00:02:19.987069 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:19.987026 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:02:19.987668 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:19.987161 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77hgn" podUID="46aec834-30bb-4d73-b1b7-f033193e6e99" Jul 30 00:02:20.130221 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:20.129999 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" event={"ID":"29dfb1a0-5edf-449a-aaaa-5f46f319fba1","Type":"ContainerStarted","Data":"19ae99944e4663b3f0162419105ce77b444af0ecbc271a30f2c90ef1a863d33c"} Jul 30 00:02:20.130623 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:20.130354 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:02:20.146610 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:20.146580 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:02:20.159225 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:20.159174 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" podStartSLOduration=8.350056197 podStartE2EDuration="27.159156889s" totalImagesPullingTime="18.809100692s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:02:20.15851638 +0000 UTC m=+27.661774640" watchObservedRunningTime="2026-07-30 00:02:20.159156889 +0000 UTC m=+27.662415150" Jul 30 00:02:20.987834 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:20.987615 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:02:20.988440 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:20.987675 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:02:20.988440 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:20.987938 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpx87" podUID="375724f6-f976-4dc0-a53a-a50fca8d270c" Jul 30 00:02:20.988440 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:20.987985 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nk989" podUID="a38e0c1c-7133-40ff-9ba4-9a066da5c41e" Jul 30 00:02:21.134533 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:21.134494 2572 generic.go:356] "Generic (PLEG): container finished" podID="5bfc194e-138e-4335-87f4-f6837c8f2e49" containerID="449396f8f7b5f55ed646989f72ca00fab876300d2ef98eb34c445cfe744a7079" exitCode=0 Jul 30 00:02:21.134718 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:21.134537 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lg8hw" event={"ID":"5bfc194e-138e-4335-87f4-f6837c8f2e49","Type":"ContainerDied","Data":"449396f8f7b5f55ed646989f72ca00fab876300d2ef98eb34c445cfe744a7079"} Jul 30 00:02:21.135705 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:21.135000 2572 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 30 00:02:21.137823 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:21.137269 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:02:21.154552 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:21.154517 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:02:21.987172 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:21.987081 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:02:21.987590 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:21.987212 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77hgn" podUID="46aec834-30bb-4d73-b1b7-f033193e6e99" Jul 30 00:02:22.140304 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:22.140141 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lg8hw" event={"ID":"5bfc194e-138e-4335-87f4-f6837c8f2e49","Type":"ContainerStarted","Data":"ab49158b858d47b2021bf81aad53fe44354dadb86d12024d94dfb5f62c0a86db"} Jul 30 00:02:22.140564 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:22.140252 2572 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 30 00:02:22.308184 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:22.308081 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-77hgn"] Jul 30 00:02:22.308448 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:22.308224 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:02:22.308736 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:22.308372 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77hgn" podUID="46aec834-30bb-4d73-b1b7-f033193e6e99" Jul 30 00:02:22.309820 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:22.309796 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nk989"] Jul 30 00:02:22.309937 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:22.309904 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:02:22.310043 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:22.310003 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nk989" podUID="a38e0c1c-7133-40ff-9ba4-9a066da5c41e" Jul 30 00:02:22.310510 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:22.310486 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xpx87"] Jul 30 00:02:22.310617 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:22.310577 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:02:22.310686 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:22.310671 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpx87" podUID="375724f6-f976-4dc0-a53a-a50fca8d270c" Jul 30 00:02:23.149501 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:23.149468 2572 generic.go:356] "Generic (PLEG): container finished" podID="5bfc194e-138e-4335-87f4-f6837c8f2e49" containerID="ab49158b858d47b2021bf81aad53fe44354dadb86d12024d94dfb5f62c0a86db" exitCode=0 Jul 30 00:02:23.150003 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:23.149502 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lg8hw" event={"ID":"5bfc194e-138e-4335-87f4-f6837c8f2e49","Type":"ContainerDied","Data":"ab49158b858d47b2021bf81aad53fe44354dadb86d12024d94dfb5f62c0a86db"} Jul 30 00:02:23.150003 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:23.149638 2572 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 30 00:02:23.987264 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:23.987175 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:02:23.987442 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:23.987175 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:02:23.987442 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:23.987308 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpx87" podUID="375724f6-f976-4dc0-a53a-a50fca8d270c" Jul 30 00:02:23.987442 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:23.987175 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:02:23.987442 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:23.987375 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nk989" podUID="a38e0c1c-7133-40ff-9ba4-9a066da5c41e" Jul 30 00:02:23.987442 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:23.987428 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77hgn" podUID="46aec834-30bb-4d73-b1b7-f033193e6e99" Jul 30 00:02:24.932467 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:24.932373 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:02:25.156397 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:25.156364 2572 generic.go:356] "Generic (PLEG): container finished" podID="5bfc194e-138e-4335-87f4-f6837c8f2e49" containerID="ad532795c1f95554222cd1a868f7ea508f1d632a16c3d63feea5bc510847bbfd" exitCode=0 Jul 30 00:02:25.156585 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:25.156424 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lg8hw" event={"ID":"5bfc194e-138e-4335-87f4-f6837c8f2e49","Type":"ContainerDied","Data":"ad532795c1f95554222cd1a868f7ea508f1d632a16c3d63feea5bc510847bbfd"} Jul 30 00:02:25.986700 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:25.986590 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:02:25.986700 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:25.986590 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:02:25.987225 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:25.986598 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:02:25.987225 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:25.987085 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nk989" podUID="a38e0c1c-7133-40ff-9ba4-9a066da5c41e" Jul 30 00:02:25.987331 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:25.987286 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpx87" podUID="375724f6-f976-4dc0-a53a-a50fca8d270c" Jul 30 00:02:25.987420 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:25.987401 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77hgn" podUID="46aec834-30bb-4d73-b1b7-f033193e6e99" Jul 30 00:02:26.628935 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:26.628892 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs\") pod \"network-metrics-daemon-nk989\" (UID: \"a38e0c1c-7133-40ff-9ba4-9a066da5c41e\") " pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:02:26.629138 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:26.629079 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 00:02:26.629185 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:26.629175 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs podName:a38e0c1c-7133-40ff-9ba4-9a066da5c41e nodeName:}" failed. No retries permitted until 2026-07-30 00:02:58.629152274 +0000 UTC m=+66.132410535 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs") pod "network-metrics-daemon-nk989" (UID: "a38e0c1c-7133-40ff-9ba4-9a066da5c41e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 00:02:26.729499 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:26.729466 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5mhv\" (UniqueName: \"kubernetes.io/projected/375724f6-f976-4dc0-a53a-a50fca8d270c-kube-api-access-m5mhv\") pod \"network-check-target-xpx87\" (UID: \"375724f6-f976-4dc0-a53a-a50fca8d270c\") " pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:02:26.729690 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:26.729531 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46aec834-30bb-4d73-b1b7-f033193e6e99-original-pull-secret\") pod \"global-pull-secret-syncer-77hgn\" (UID: \"46aec834-30bb-4d73-b1b7-f033193e6e99\") " pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:02:26.729690 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:26.729610 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 00:02:26.729690 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:26.729630 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 00:02:26.729690 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:26.729644 2572 projected.go:196] Error preparing data for projected volume kube-api-access-m5mhv for pod openshift-network-diagnostics/network-check-target-xpx87: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 00:02:26.729690 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:26.729651 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 00:02:26.729951 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:26.729718 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/375724f6-f976-4dc0-a53a-a50fca8d270c-kube-api-access-m5mhv podName:375724f6-f976-4dc0-a53a-a50fca8d270c nodeName:}" failed. No retries permitted until 2026-07-30 00:02:58.729700041 +0000 UTC m=+66.232958284 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-m5mhv" (UniqueName: "kubernetes.io/projected/375724f6-f976-4dc0-a53a-a50fca8d270c-kube-api-access-m5mhv") pod "network-check-target-xpx87" (UID: "375724f6-f976-4dc0-a53a-a50fca8d270c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 00:02:26.729951 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:26.729735 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46aec834-30bb-4d73-b1b7-f033193e6e99-original-pull-secret podName:46aec834-30bb-4d73-b1b7-f033193e6e99 nodeName:}" failed. No retries permitted until 2026-07-30 00:02:58.729727172 +0000 UTC m=+66.232985414 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46aec834-30bb-4d73-b1b7-f033193e6e99-original-pull-secret") pod "global-pull-secret-syncer-77hgn" (UID: "46aec834-30bb-4d73-b1b7-f033193e6e99") : object "kube-system"/"original-pull-secret" not registered Jul 30 00:02:27.345323 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.345291 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-118.ec2.internal" event="NodeReady" Jul 30 00:02:27.345891 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.345455 2572 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 30 00:02:27.387362 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.387329 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tb6c5"] Jul 30 00:02:27.389466 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.389445 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tb6c5" Jul 30 00:02:27.392432 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.392406 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 30 00:02:27.392560 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.392516 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nxhcc\"" Jul 30 00:02:27.392560 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.392516 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 30 00:02:27.393365 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.393343 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mksd5"] Jul 30 00:02:27.395381 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.395359 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mksd5" Jul 30 00:02:27.398795 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.398549 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 30 00:02:27.398795 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.398617 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 30 00:02:27.398939 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.398832 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 30 00:02:27.398939 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.398831 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-559dl\"" Jul 30 00:02:27.401304 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.401282 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tb6c5"] Jul 30 00:02:27.407304 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.407277 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mksd5"] Jul 30 00:02:27.469736 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.469646 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:02:27.536278 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.536241 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9b8edba-eb6a-4906-97a4-4eb273367aed-cert\") pod \"ingress-canary-mksd5\" (UID: \"b9b8edba-eb6a-4906-97a4-4eb273367aed\") " pod="openshift-ingress-canary/ingress-canary-mksd5" Jul 30 00:02:27.536468 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.536309 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkmct\" (UniqueName: \"kubernetes.io/projected/b9b8edba-eb6a-4906-97a4-4eb273367aed-kube-api-access-vkmct\") pod \"ingress-canary-mksd5\" (UID: \"b9b8edba-eb6a-4906-97a4-4eb273367aed\") " pod="openshift-ingress-canary/ingress-canary-mksd5" Jul 30 00:02:27.536468 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.536364 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls\") pod \"dns-default-tb6c5\" (UID: \"34dd2a14-f850-4a05-ab6b-9c0c0196ed0a\") " pod="openshift-dns/dns-default-tb6c5" Jul 30 00:02:27.536468 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.536392 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxh8b\" (UniqueName: \"kubernetes.io/projected/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-kube-api-access-xxh8b\") pod \"dns-default-tb6c5\" (UID: \"34dd2a14-f850-4a05-ab6b-9c0c0196ed0a\") " pod="openshift-dns/dns-default-tb6c5" Jul 30 00:02:27.536468 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.536420 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-config-volume\") pod \"dns-default-tb6c5\" (UID: \"34dd2a14-f850-4a05-ab6b-9c0c0196ed0a\") " pod="openshift-dns/dns-default-tb6c5" Jul 30 00:02:27.536468 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.536446 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-tmp-dir\") pod \"dns-default-tb6c5\" (UID: \"34dd2a14-f850-4a05-ab6b-9c0c0196ed0a\") " pod="openshift-dns/dns-default-tb6c5" Jul 30 00:02:27.552144 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.552103 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:02:27.637027 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.636990 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9b8edba-eb6a-4906-97a4-4eb273367aed-cert\") pod \"ingress-canary-mksd5\" (UID: \"b9b8edba-eb6a-4906-97a4-4eb273367aed\") " pod="openshift-ingress-canary/ingress-canary-mksd5" Jul 30 00:02:27.637242 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.637056 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkmct\" (UniqueName: \"kubernetes.io/projected/b9b8edba-eb6a-4906-97a4-4eb273367aed-kube-api-access-vkmct\") pod \"ingress-canary-mksd5\" (UID: \"b9b8edba-eb6a-4906-97a4-4eb273367aed\") " pod="openshift-ingress-canary/ingress-canary-mksd5" Jul 30 00:02:27.637242 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.637087 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls\") pod \"dns-default-tb6c5\" (UID: \"34dd2a14-f850-4a05-ab6b-9c0c0196ed0a\") " pod="openshift-dns/dns-default-tb6c5" Jul 30 00:02:27.637242 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.637136 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxh8b\" (UniqueName: \"kubernetes.io/projected/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-kube-api-access-xxh8b\") pod \"dns-default-tb6c5\" (UID: \"34dd2a14-f850-4a05-ab6b-9c0c0196ed0a\") " pod="openshift-dns/dns-default-tb6c5" Jul 30 00:02:27.637242 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.637197 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-config-volume\") pod \"dns-default-tb6c5\" (UID: \"34dd2a14-f850-4a05-ab6b-9c0c0196ed0a\") " pod="openshift-dns/dns-default-tb6c5" Jul 30 00:02:27.637242 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.637236 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-tmp-dir\") pod \"dns-default-tb6c5\" (UID: \"34dd2a14-f850-4a05-ab6b-9c0c0196ed0a\") " pod="openshift-dns/dns-default-tb6c5" Jul 30 00:02:27.637507 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:27.637320 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 00:02:27.637507 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:27.637380 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls podName:34dd2a14-f850-4a05-ab6b-9c0c0196ed0a nodeName:}" failed. No retries permitted until 2026-07-30 00:02:28.137358771 +0000 UTC m=+35.640617012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls") pod "dns-default-tb6c5" (UID: "34dd2a14-f850-4a05-ab6b-9c0c0196ed0a") : secret "dns-default-metrics-tls" not found Jul 30 00:02:27.637817 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.637798 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-config-volume\") pod \"dns-default-tb6c5\" (UID: \"34dd2a14-f850-4a05-ab6b-9c0c0196ed0a\") " pod="openshift-dns/dns-default-tb6c5" Jul 30 00:02:27.638292 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.638270 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-tmp-dir\") pod \"dns-default-tb6c5\" (UID: \"34dd2a14-f850-4a05-ab6b-9c0c0196ed0a\") " pod="openshift-dns/dns-default-tb6c5" Jul 30 00:02:27.642187 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.642148 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9b8edba-eb6a-4906-97a4-4eb273367aed-cert\") pod \"ingress-canary-mksd5\" (UID: \"b9b8edba-eb6a-4906-97a4-4eb273367aed\") " pod="openshift-ingress-canary/ingress-canary-mksd5" Jul 30 00:02:27.648323 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.648042 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxh8b\" (UniqueName: \"kubernetes.io/projected/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-kube-api-access-xxh8b\") pod \"dns-default-tb6c5\" (UID: \"34dd2a14-f850-4a05-ab6b-9c0c0196ed0a\") " pod="openshift-dns/dns-default-tb6c5" Jul 30 00:02:27.648323 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.648280 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkmct\" (UniqueName: \"kubernetes.io/projected/b9b8edba-eb6a-4906-97a4-4eb273367aed-kube-api-access-vkmct\") pod \"ingress-canary-mksd5\" (UID: \"b9b8edba-eb6a-4906-97a4-4eb273367aed\") " pod="openshift-ingress-canary/ingress-canary-mksd5" Jul 30 00:02:27.709271 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.709235 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mksd5" Jul 30 00:02:27.867287 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.867242 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mksd5"] Jul 30 00:02:27.871305 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:02:27.871272 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b8edba_eb6a_4906_97a4_4eb273367aed.slice/crio-c82463ee52877f3808f211898cd3b85d7ac6ca9872b7902f22816b41191974c3 WatchSource:0}: Error finding container c82463ee52877f3808f211898cd3b85d7ac6ca9872b7902f22816b41191974c3: Status 404 returned error can't find the container with id c82463ee52877f3808f211898cd3b85d7ac6ca9872b7902f22816b41191974c3 Jul 30 00:02:27.987302 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.987194 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:02:27.987497 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.987302 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:02:27.987497 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.987321 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:02:27.990422 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.990395 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 30 00:02:27.990566 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.990450 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 30 00:02:27.990566 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.990397 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 30 00:02:27.990566 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.990398 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 30 00:02:27.991630 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.991609 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-p64v7\"" Jul 30 00:02:27.991630 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:27.991623 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qpk4b\"" Jul 30 00:02:28.143640 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:28.143588 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls\") pod \"dns-default-tb6c5\" (UID: \"34dd2a14-f850-4a05-ab6b-9c0c0196ed0a\") " pod="openshift-dns/dns-default-tb6c5" Jul 30 00:02:28.143858 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:28.143810 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 00:02:28.143958 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:28.143942 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls podName:34dd2a14-f850-4a05-ab6b-9c0c0196ed0a nodeName:}" failed. No retries permitted until 2026-07-30 00:02:29.143909297 +0000 UTC m=+36.647167556 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls") pod "dns-default-tb6c5" (UID: "34dd2a14-f850-4a05-ab6b-9c0c0196ed0a") : secret "dns-default-metrics-tls" not found Jul 30 00:02:28.165367 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:28.165328 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mksd5" event={"ID":"b9b8edba-eb6a-4906-97a4-4eb273367aed","Type":"ContainerStarted","Data":"c82463ee52877f3808f211898cd3b85d7ac6ca9872b7902f22816b41191974c3"} Jul 30 00:02:28.425391 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:28.425353 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:02:28.485327 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:28.485285 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:02:28.843261 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:28.843227 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:02:29.152727 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:29.152636 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls\") pod \"dns-default-tb6c5\" (UID: \"34dd2a14-f850-4a05-ab6b-9c0c0196ed0a\") " pod="openshift-dns/dns-default-tb6c5" Jul 30 00:02:29.152883 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:29.152804 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 00:02:29.152883 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:29.152869 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls podName:34dd2a14-f850-4a05-ab6b-9c0c0196ed0a nodeName:}" failed. No retries permitted until 2026-07-30 00:02:31.152852448 +0000 UTC m=+38.656110688 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls") pod "dns-default-tb6c5" (UID: "34dd2a14-f850-4a05-ab6b-9c0c0196ed0a") : secret "dns-default-metrics-tls" not found Jul 30 00:02:29.495019 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:29.494937 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:02:29.581120 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:29.581068 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:02:30.830457 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:30.830251 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:02:31.169025 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:31.168946 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls\") pod \"dns-default-tb6c5\" (UID: \"34dd2a14-f850-4a05-ab6b-9c0c0196ed0a\") " pod="openshift-dns/dns-default-tb6c5" Jul 30 00:02:31.169184 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:31.169071 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 00:02:31.169184 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:31.169159 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls podName:34dd2a14-f850-4a05-ab6b-9c0c0196ed0a nodeName:}" failed. No retries permitted until 2026-07-30 00:02:35.169138233 +0000 UTC m=+42.672396473 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls") pod "dns-default-tb6c5" (UID: "34dd2a14-f850-4a05-ab6b-9c0c0196ed0a") : secret "dns-default-metrics-tls" not found Jul 30 00:02:31.177129 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:31.177068 2572 generic.go:356] "Generic (PLEG): container finished" podID="5bfc194e-138e-4335-87f4-f6837c8f2e49" containerID="963b60f244f5995ae568e013e2cb1eadf49dfd600ce63784dddd7128bf053ee6" exitCode=0 Jul 30 00:02:31.177288 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:31.177151 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lg8hw" event={"ID":"5bfc194e-138e-4335-87f4-f6837c8f2e49","Type":"ContainerDied","Data":"963b60f244f5995ae568e013e2cb1eadf49dfd600ce63784dddd7128bf053ee6"} Jul 30 00:02:31.709240 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:31.709197 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:02:31.762475 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:31.762435 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:02:31.889250 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:31.889090 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:02:31.990529 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:31.990431 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:02:32.182513 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:32.182479 2572 generic.go:356] "Generic (PLEG): container finished" podID="5bfc194e-138e-4335-87f4-f6837c8f2e49" containerID="f415a905d06fe5c45712b4429ce0cee00583eeb54a40ad7b0bddeff642b9f7c2" exitCode=0 Jul 30 00:02:32.182679 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:32.182528 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lg8hw" event={"ID":"5bfc194e-138e-4335-87f4-f6837c8f2e49","Type":"ContainerDied","Data":"f415a905d06fe5c45712b4429ce0cee00583eeb54a40ad7b0bddeff642b9f7c2"} Jul 30 00:02:33.186349 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:33.186307 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mksd5" event={"ID":"b9b8edba-eb6a-4906-97a4-4eb273367aed","Type":"ContainerStarted","Data":"645f6d3e06ae6bec42c4c92cc7f1917195e24e5c0140c8eb99c04cd0f0e40b88"} Jul 30 00:02:33.189796 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:33.189761 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lg8hw" event={"ID":"5bfc194e-138e-4335-87f4-f6837c8f2e49","Type":"ContainerStarted","Data":"dab8c45ba08b13d10c5f4f90e20326d158b41fbb5103282f79238daf85268dae"} Jul 30 00:02:33.200727 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:33.200662 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mksd5" podStartSLOduration=1.484503076 podStartE2EDuration="6.200643577s" totalImagesPullingTime="4.716140501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:02:33.199974021 +0000 UTC m=+40.703232280" watchObservedRunningTime="2026-07-30 00:02:33.200643577 +0000 UTC m=+40.703901839" Jul 30 00:02:33.220904 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:33.220845 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lg8hw" podStartSLOduration=9.263227418 podStartE2EDuration="40.220826958s" totalImagesPullingTime="30.95759954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:01:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:02:33.219398713 +0000 UTC m=+40.722656973" watchObservedRunningTime="2026-07-30 00:02:33.220826958 +0000 UTC m=+40.724085218" Jul 30 00:02:33.346983 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:33.346947 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:02:33.415268 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:33.415233 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:02:35.200412 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:35.200377 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls\") pod \"dns-default-tb6c5\" (UID: \"34dd2a14-f850-4a05-ab6b-9c0c0196ed0a\") " pod="openshift-dns/dns-default-tb6c5" Jul 30 00:02:35.200801 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:35.200524 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 00:02:35.200801 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:35.200590 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls podName:34dd2a14-f850-4a05-ab6b-9c0c0196ed0a nodeName:}" failed. No retries permitted until 2026-07-30 00:02:43.200568028 +0000 UTC m=+50.703826267 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls") pod "dns-default-tb6c5" (UID: "34dd2a14-f850-4a05-ab6b-9c0c0196ed0a") : secret "dns-default-metrics-tls" not found Jul 30 00:02:36.389531 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:36.389491 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:02:36.475481 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:36.475447 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:02:40.165464 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:40.165430 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:02:40.212667 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:40.212637 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:02:43.255127 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:43.255068 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls\") pod \"dns-default-tb6c5\" (UID: \"34dd2a14-f850-4a05-ab6b-9c0c0196ed0a\") " pod="openshift-dns/dns-default-tb6c5" Jul 30 00:02:43.255512 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:43.255218 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 00:02:43.255512 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:43.255292 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls podName:34dd2a14-f850-4a05-ab6b-9c0c0196ed0a nodeName:}" failed. No retries permitted until 2026-07-30 00:02:59.255262994 +0000 UTC m=+66.758521235 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls") pod "dns-default-tb6c5" (UID: "34dd2a14-f850-4a05-ab6b-9c0c0196ed0a") : secret "dns-default-metrics-tls" not found Jul 30 00:02:44.559259 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:44.559225 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:02:44.660210 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:44.660175 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:02:44.839324 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:44.839295 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:02:44.940760 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:44.940713 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:02:52.698905 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:52.698870 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:02:52.799722 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:52.799681 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:02:54.943546 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:54.943517 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l48hb" Jul 30 00:02:55.090198 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:55.090162 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:02:55.190583 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:55.190549 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:02:58.666802 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:58.666745 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs\") pod \"network-metrics-daemon-nk989\" (UID: \"a38e0c1c-7133-40ff-9ba4-9a066da5c41e\") " pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:02:58.669580 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:58.669561 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 30 00:02:58.677613 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:58.677587 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 30 00:02:58.677672 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:58.677661 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs podName:a38e0c1c-7133-40ff-9ba4-9a066da5c41e nodeName:}" failed. No retries permitted until 2026-07-30 00:04:02.67763934 +0000 UTC m=+130.180897578 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs") pod "network-metrics-daemon-nk989" (UID: "a38e0c1c-7133-40ff-9ba4-9a066da5c41e") : secret "metrics-daemon-secret" not found Jul 30 00:02:58.767088 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:58.767052 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5mhv\" (UniqueName: \"kubernetes.io/projected/375724f6-f976-4dc0-a53a-a50fca8d270c-kube-api-access-m5mhv\") pod \"network-check-target-xpx87\" (UID: \"375724f6-f976-4dc0-a53a-a50fca8d270c\") " pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:02:58.767204 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:58.767103 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46aec834-30bb-4d73-b1b7-f033193e6e99-original-pull-secret\") pod \"global-pull-secret-syncer-77hgn\" (UID: \"46aec834-30bb-4d73-b1b7-f033193e6e99\") " pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:02:58.769868 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:58.769847 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 30 00:02:58.770003 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:58.769990 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 30 00:02:58.780083 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:58.780053 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 30 00:02:58.781664 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:58.781644 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46aec834-30bb-4d73-b1b7-f033193e6e99-original-pull-secret\") pod \"global-pull-secret-syncer-77hgn\" (UID: \"46aec834-30bb-4d73-b1b7-f033193e6e99\") " pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:02:58.790468 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:58.790446 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5mhv\" (UniqueName: \"kubernetes.io/projected/375724f6-f976-4dc0-a53a-a50fca8d270c-kube-api-access-m5mhv\") pod \"network-check-target-xpx87\" (UID: \"375724f6-f976-4dc0-a53a-a50fca8d270c\") " pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:02:58.900865 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:58.900810 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77hgn" Jul 30 00:02:58.917643 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:58.917583 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-p64v7\"" Jul 30 00:02:58.925611 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:58.925587 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:02:59.033189 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:59.033035 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-77hgn"] Jul 30 00:02:59.035740 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:02:59.035705 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46aec834_30bb_4d73_b1b7_f033193e6e99.slice/crio-9c52fd1e7e4ba5b91a3039d48c1d17c8dda72051b9e6bae0d39f5ae2f739d0e3 WatchSource:0}: Error finding container 9c52fd1e7e4ba5b91a3039d48c1d17c8dda72051b9e6bae0d39f5ae2f739d0e3: Status 404 returned error can't find the container with id 9c52fd1e7e4ba5b91a3039d48c1d17c8dda72051b9e6bae0d39f5ae2f739d0e3 Jul 30 00:02:59.051308 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:59.051268 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xpx87"] Jul 30 00:02:59.053988 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:02:59.053963 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod375724f6_f976_4dc0_a53a_a50fca8d270c.slice/crio-9a4d397aca3db78ee6f5d75e08df9bbbac281d7ebb0cf5b0c240c68f4da1d479 WatchSource:0}: Error finding container 9a4d397aca3db78ee6f5d75e08df9bbbac281d7ebb0cf5b0c240c68f4da1d479: Status 404 returned error can't find the container with id 9a4d397aca3db78ee6f5d75e08df9bbbac281d7ebb0cf5b0c240c68f4da1d479 Jul 30 00:02:59.252906 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:59.252812 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xpx87" event={"ID":"375724f6-f976-4dc0-a53a-a50fca8d270c","Type":"ContainerStarted","Data":"9a4d397aca3db78ee6f5d75e08df9bbbac281d7ebb0cf5b0c240c68f4da1d479"} Jul 30 00:02:59.253978 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:59.253945 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-77hgn" event={"ID":"46aec834-30bb-4d73-b1b7-f033193e6e99","Type":"ContainerStarted","Data":"9c52fd1e7e4ba5b91a3039d48c1d17c8dda72051b9e6bae0d39f5ae2f739d0e3"} Jul 30 00:02:59.270388 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:02:59.270353 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls\") pod \"dns-default-tb6c5\" (UID: \"34dd2a14-f850-4a05-ab6b-9c0c0196ed0a\") " pod="openshift-dns/dns-default-tb6c5" Jul 30 00:02:59.270483 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:59.270466 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 00:02:59.270542 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:02:59.270532 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls podName:34dd2a14-f850-4a05-ab6b-9c0c0196ed0a nodeName:}" failed. No retries permitted until 2026-07-30 00:03:31.270516107 +0000 UTC m=+98.773774348 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls") pod "dns-default-tb6c5" (UID: "34dd2a14-f850-4a05-ab6b-9c0c0196ed0a") : secret "dns-default-metrics-tls" not found Jul 30 00:03:03.266985 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:03.266944 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xpx87" event={"ID":"375724f6-f976-4dc0-a53a-a50fca8d270c","Type":"ContainerStarted","Data":"898afd1fd60f66bb4091e2ebc3a4d63c901f9354dd0e8ae31494070142bdd1b3"} Jul 30 00:03:03.267469 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:03.267064 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:03:03.282439 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:03.282373 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xpx87" podStartSLOduration=67.111271101 podStartE2EDuration="1m10.282355342s" totalImagesPullingTime="3.171084241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:03:03.28156775 +0000 UTC m=+70.784826011" watchObservedRunningTime="2026-07-30 00:03:03.282355342 +0000 UTC m=+70.785613601" Jul 30 00:03:04.270690 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:04.270651 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-77hgn" event={"ID":"46aec834-30bb-4d73-b1b7-f033193e6e99","Type":"ContainerStarted","Data":"b9743b1490ca530411427ea94c3b6543f45860cd2e1af940a3ae0bfaae1589ce"} Jul 30 00:03:04.285637 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:04.285567 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-77hgn" podStartSLOduration=66.529896555 podStartE2EDuration="1m11.285548492s" totalImagesPullingTime="4.755651937s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:03:04.285165165 +0000 UTC m=+71.788423425" watchObservedRunningTime="2026-07-30 00:03:04.285548492 +0000 UTC m=+71.788806753" Jul 30 00:03:06.513223 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:06.513178 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:03:06.616079 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:06.616043 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:03:31.286185 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.286148 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls\") pod \"dns-default-tb6c5\" (UID: \"34dd2a14-f850-4a05-ab6b-9c0c0196ed0a\") " pod="openshift-dns/dns-default-tb6c5" Jul 30 00:03:31.286586 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:03:31.286268 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 00:03:31.286586 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:03:31.286338 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls podName:34dd2a14-f850-4a05-ab6b-9c0c0196ed0a nodeName:}" failed. No retries permitted until 2026-07-30 00:04:35.28632295 +0000 UTC m=+162.789581188 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls") pod "dns-default-tb6c5" (UID: "34dd2a14-f850-4a05-ab6b-9c0c0196ed0a") : secret "dns-default-metrics-tls" not found Jul 30 00:03:31.506681 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.506652 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-864dbcc49f-wccmz"] Jul 30 00:03:31.510749 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.510733 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864dbcc49f-wccmz" Jul 30 00:03:31.513069 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.513047 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 30 00:03:31.513206 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.513174 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 30 00:03:31.514426 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.514409 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 30 00:03:31.514511 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.514480 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 30 00:03:31.518882 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.518862 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-864dbcc49f-wccmz"] Jul 30 00:03:31.553982 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.553920 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp"] Jul 30 00:03:31.556654 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.556640 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" Jul 30 00:03:31.559534 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.559513 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 30 00:03:31.559809 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.559791 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 30 00:03:31.559937 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.559920 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 30 00:03:31.560513 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.560497 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 30 00:03:31.567001 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.566979 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp"] Jul 30 00:03:31.588414 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.588389 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/86def107-a402-4f54-a076-c26bcf2c0d7a-klusterlet-config\") pod \"klusterlet-addon-workmgr-864dbcc49f-wccmz\" (UID: \"86def107-a402-4f54-a076-c26bcf2c0d7a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864dbcc49f-wccmz" Jul 30 00:03:31.588517 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.588421 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7npm\" (UniqueName: \"kubernetes.io/projected/86def107-a402-4f54-a076-c26bcf2c0d7a-kube-api-access-s7npm\") pod \"klusterlet-addon-workmgr-864dbcc49f-wccmz\" (UID: \"86def107-a402-4f54-a076-c26bcf2c0d7a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864dbcc49f-wccmz" Jul 30 00:03:31.588517 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.588451 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86def107-a402-4f54-a076-c26bcf2c0d7a-tmp\") pod \"klusterlet-addon-workmgr-864dbcc49f-wccmz\" (UID: \"86def107-a402-4f54-a076-c26bcf2c0d7a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864dbcc49f-wccmz" Jul 30 00:03:31.688981 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.688947 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/86def107-a402-4f54-a076-c26bcf2c0d7a-klusterlet-config\") pod \"klusterlet-addon-workmgr-864dbcc49f-wccmz\" (UID: \"86def107-a402-4f54-a076-c26bcf2c0d7a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864dbcc49f-wccmz" Jul 30 00:03:31.688981 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.688983 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6fc6f82-c5da-479b-a03b-6022fc8d678f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5cc8795f79-hswhp\" (UID: \"d6fc6f82-c5da-479b-a03b-6022fc8d678f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" Jul 30 00:03:31.689194 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.689004 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7npm\" (UniqueName: \"kubernetes.io/projected/86def107-a402-4f54-a076-c26bcf2c0d7a-kube-api-access-s7npm\") pod \"klusterlet-addon-workmgr-864dbcc49f-wccmz\" (UID: \"86def107-a402-4f54-a076-c26bcf2c0d7a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864dbcc49f-wccmz" Jul 30 00:03:31.689194 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.689020 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d6fc6f82-c5da-479b-a03b-6022fc8d678f-ca\") pod \"cluster-proxy-proxy-agent-5cc8795f79-hswhp\" (UID: \"d6fc6f82-c5da-479b-a03b-6022fc8d678f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" Jul 30 00:03:31.689194 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.689036 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8ftt\" (UniqueName: \"kubernetes.io/projected/d6fc6f82-c5da-479b-a03b-6022fc8d678f-kube-api-access-s8ftt\") pod \"cluster-proxy-proxy-agent-5cc8795f79-hswhp\" (UID: \"d6fc6f82-c5da-479b-a03b-6022fc8d678f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" Jul 30 00:03:31.689194 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.689097 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86def107-a402-4f54-a076-c26bcf2c0d7a-tmp\") pod \"klusterlet-addon-workmgr-864dbcc49f-wccmz\" (UID: \"86def107-a402-4f54-a076-c26bcf2c0d7a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864dbcc49f-wccmz" Jul 30 00:03:31.689194 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.689151 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6fc6f82-c5da-479b-a03b-6022fc8d678f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5cc8795f79-hswhp\" (UID: \"d6fc6f82-c5da-479b-a03b-6022fc8d678f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" Jul 30 00:03:31.689194 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.689181 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d6fc6f82-c5da-479b-a03b-6022fc8d678f-hub\") pod \"cluster-proxy-proxy-agent-5cc8795f79-hswhp\" (UID: \"d6fc6f82-c5da-479b-a03b-6022fc8d678f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" Jul 30 00:03:31.689397 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.689201 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d6fc6f82-c5da-479b-a03b-6022fc8d678f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5cc8795f79-hswhp\" (UID: \"d6fc6f82-c5da-479b-a03b-6022fc8d678f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" Jul 30 00:03:31.689515 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.689492 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86def107-a402-4f54-a076-c26bcf2c0d7a-tmp\") pod \"klusterlet-addon-workmgr-864dbcc49f-wccmz\" (UID: \"86def107-a402-4f54-a076-c26bcf2c0d7a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864dbcc49f-wccmz" Jul 30 00:03:31.691443 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.691419 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/86def107-a402-4f54-a076-c26bcf2c0d7a-klusterlet-config\") pod \"klusterlet-addon-workmgr-864dbcc49f-wccmz\" (UID: \"86def107-a402-4f54-a076-c26bcf2c0d7a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864dbcc49f-wccmz" Jul 30 00:03:31.709136 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.709099 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7npm\" (UniqueName: \"kubernetes.io/projected/86def107-a402-4f54-a076-c26bcf2c0d7a-kube-api-access-s7npm\") pod \"klusterlet-addon-workmgr-864dbcc49f-wccmz\" (UID: \"86def107-a402-4f54-a076-c26bcf2c0d7a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864dbcc49f-wccmz" Jul 30 00:03:31.790242 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.790203 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d6fc6f82-c5da-479b-a03b-6022fc8d678f-hub\") pod \"cluster-proxy-proxy-agent-5cc8795f79-hswhp\" (UID: \"d6fc6f82-c5da-479b-a03b-6022fc8d678f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" Jul 30 00:03:31.790405 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.790250 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d6fc6f82-c5da-479b-a03b-6022fc8d678f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5cc8795f79-hswhp\" (UID: \"d6fc6f82-c5da-479b-a03b-6022fc8d678f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" Jul 30 00:03:31.790405 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.790314 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6fc6f82-c5da-479b-a03b-6022fc8d678f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5cc8795f79-hswhp\" (UID: \"d6fc6f82-c5da-479b-a03b-6022fc8d678f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" Jul 30 00:03:31.790405 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.790336 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d6fc6f82-c5da-479b-a03b-6022fc8d678f-ca\") pod \"cluster-proxy-proxy-agent-5cc8795f79-hswhp\" (UID: \"d6fc6f82-c5da-479b-a03b-6022fc8d678f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" Jul 30 00:03:31.790405 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.790358 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8ftt\" (UniqueName: \"kubernetes.io/projected/d6fc6f82-c5da-479b-a03b-6022fc8d678f-kube-api-access-s8ftt\") pod \"cluster-proxy-proxy-agent-5cc8795f79-hswhp\" (UID: \"d6fc6f82-c5da-479b-a03b-6022fc8d678f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" Jul 30 00:03:31.790405 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.790399 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6fc6f82-c5da-479b-a03b-6022fc8d678f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5cc8795f79-hswhp\" (UID: \"d6fc6f82-c5da-479b-a03b-6022fc8d678f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" Jul 30 00:03:31.791204 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.791175 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6fc6f82-c5da-479b-a03b-6022fc8d678f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5cc8795f79-hswhp\" (UID: \"d6fc6f82-c5da-479b-a03b-6022fc8d678f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" Jul 30 00:03:31.792759 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.792726 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d6fc6f82-c5da-479b-a03b-6022fc8d678f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5cc8795f79-hswhp\" (UID: \"d6fc6f82-c5da-479b-a03b-6022fc8d678f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" Jul 30 00:03:31.792876 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.792799 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d6fc6f82-c5da-479b-a03b-6022fc8d678f-ca\") pod \"cluster-proxy-proxy-agent-5cc8795f79-hswhp\" (UID: \"d6fc6f82-c5da-479b-a03b-6022fc8d678f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" Jul 30 00:03:31.792961 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.792943 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6fc6f82-c5da-479b-a03b-6022fc8d678f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5cc8795f79-hswhp\" (UID: \"d6fc6f82-c5da-479b-a03b-6022fc8d678f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" Jul 30 00:03:31.793024 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.793004 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d6fc6f82-c5da-479b-a03b-6022fc8d678f-hub\") pod \"cluster-proxy-proxy-agent-5cc8795f79-hswhp\" (UID: \"d6fc6f82-c5da-479b-a03b-6022fc8d678f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" Jul 30 00:03:31.798260 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.798237 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8ftt\" (UniqueName: \"kubernetes.io/projected/d6fc6f82-c5da-479b-a03b-6022fc8d678f-kube-api-access-s8ftt\") pod \"cluster-proxy-proxy-agent-5cc8795f79-hswhp\" (UID: \"d6fc6f82-c5da-479b-a03b-6022fc8d678f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" Jul 30 00:03:31.819996 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.819922 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864dbcc49f-wccmz" Jul 30 00:03:31.883539 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.883509 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" Jul 30 00:03:31.943802 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:31.943776 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-864dbcc49f-wccmz"] Jul 30 00:03:31.946663 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:03:31.946639 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86def107_a402_4f54_a076_c26bcf2c0d7a.slice/crio-d39c2b69ff9ebf4d8a0b78d06d31c687598f343f39152bf755ce0c87d556f590 WatchSource:0}: Error finding container d39c2b69ff9ebf4d8a0b78d06d31c687598f343f39152bf755ce0c87d556f590: Status 404 returned error can't find the container with id d39c2b69ff9ebf4d8a0b78d06d31c687598f343f39152bf755ce0c87d556f590 Jul 30 00:03:32.009213 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:32.009185 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp"] Jul 30 00:03:32.009979 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:03:32.009955 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6fc6f82_c5da_479b_a03b_6022fc8d678f.slice/crio-764788317ecf1fb4a0fcd4ae8cbf2555c0f0b3b0b1b17c233493454c64ce4d0d WatchSource:0}: Error finding container 764788317ecf1fb4a0fcd4ae8cbf2555c0f0b3b0b1b17c233493454c64ce4d0d: Status 404 returned error can't find the container with id 764788317ecf1fb4a0fcd4ae8cbf2555c0f0b3b0b1b17c233493454c64ce4d0d Jul 30 00:03:32.337438 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:32.337398 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864dbcc49f-wccmz" event={"ID":"86def107-a402-4f54-a076-c26bcf2c0d7a","Type":"ContainerStarted","Data":"d39c2b69ff9ebf4d8a0b78d06d31c687598f343f39152bf755ce0c87d556f590"} Jul 30 00:03:32.338513 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:32.338484 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" event={"ID":"d6fc6f82-c5da-479b-a03b-6022fc8d678f","Type":"ContainerStarted","Data":"764788317ecf1fb4a0fcd4ae8cbf2555c0f0b3b0b1b17c233493454c64ce4d0d"} Jul 30 00:03:34.274012 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:34.273949 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xpx87" Jul 30 00:03:38.360795 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:38.360758 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" event={"ID":"d6fc6f82-c5da-479b-a03b-6022fc8d678f","Type":"ContainerStarted","Data":"a178f843ac52271629e1049486ffa9c80ac43692cfa4eaa15d39f6ade2ddedf2"} Jul 30 00:03:38.361379 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:38.360812 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" event={"ID":"d6fc6f82-c5da-479b-a03b-6022fc8d678f","Type":"ContainerStarted","Data":"01b19b63ce15119470b5152242840e6aa532bccacf5aff3d5222b5bdb12969da"} Jul 30 00:03:38.361379 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:38.360829 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" event={"ID":"d6fc6f82-c5da-479b-a03b-6022fc8d678f","Type":"ContainerStarted","Data":"36c65e0dbfb93ae6d638e912e672ffd0b922dace015c09541c0ca0fe38ee5d15"} Jul 30 00:03:38.362080 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:38.362058 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864dbcc49f-wccmz" event={"ID":"86def107-a402-4f54-a076-c26bcf2c0d7a","Type":"ContainerStarted","Data":"52c9ecc86b1cc337b6183dafad5e17af0b79716546de80a3b2933d9c4d2f2d29"} Jul 30 00:03:38.362205 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:38.362161 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864dbcc49f-wccmz" Jul 30 00:03:38.364527 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:38.364509 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864dbcc49f-wccmz" Jul 30 00:03:38.379706 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:38.379669 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" podStartSLOduration=1.612926979 podStartE2EDuration="7.379658409s" totalImagesPullingTime="5.76673143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:03:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:03:38.379031872 +0000 UTC m=+105.882290132" watchObservedRunningTime="2026-07-30 00:03:38.379658409 +0000 UTC m=+105.882916673" Jul 30 00:03:38.395843 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:38.395795 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864dbcc49f-wccmz" podStartSLOduration=1.564045752 podStartE2EDuration="7.395781218s" totalImagesPullingTime="5.831735466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:03:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:03:38.395726461 +0000 UTC m=+105.898984720" watchObservedRunningTime="2026-07-30 00:03:38.395781218 +0000 UTC m=+105.899039478" Jul 30 00:03:39.657461 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:03:39.657429 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:04:02.711457 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:04:02.711408 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs\") pod \"network-metrics-daemon-nk989\" (UID: \"a38e0c1c-7133-40ff-9ba4-9a066da5c41e\") " pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:04:02.711859 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:04:02.711537 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 30 00:04:02.711859 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:04:02.711591 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs podName:a38e0c1c-7133-40ff-9ba4-9a066da5c41e nodeName:}" failed. No retries permitted until 2026-07-30 00:06:04.711576834 +0000 UTC m=+252.214835072 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs") pod "network-metrics-daemon-nk989" (UID: "a38e0c1c-7133-40ff-9ba4-9a066da5c41e") : secret "metrics-daemon-secret" not found Jul 30 00:04:25.438290 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:04:25.438257 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:04:30.402715 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:04:30.402656 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-tb6c5" podUID="34dd2a14-f850-4a05-ab6b-9c0c0196ed0a" Jul 30 00:04:30.509614 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:04:30.509586 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tb6c5" Jul 30 00:04:31.008346 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:04:31.008309 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-nk989" podUID="a38e0c1c-7133-40ff-9ba4-9a066da5c41e" Jul 30 00:04:35.326270 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:04:35.326223 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls\") pod \"dns-default-tb6c5\" (UID: \"34dd2a14-f850-4a05-ab6b-9c0c0196ed0a\") " pod="openshift-dns/dns-default-tb6c5" Jul 30 00:04:35.326760 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:04:35.326395 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 00:04:35.326760 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:04:35.326479 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls podName:34dd2a14-f850-4a05-ab6b-9c0c0196ed0a nodeName:}" failed. No retries permitted until 2026-07-30 00:06:37.32645783 +0000 UTC m=+284.829716074 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls") pod "dns-default-tb6c5" (UID: "34dd2a14-f850-4a05-ab6b-9c0c0196ed0a") : secret "dns-default-metrics-tls" not found Jul 30 00:04:45.987354 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:04:45.987268 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:05:38.363354 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:05:38.363302 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864dbcc49f-wccmz" podUID="86def107-a402-4f54-a076-c26bcf2c0d7a" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.8:8000/readyz\": dial tcp 10.133.0.8:8000: connect: connection refused" Jul 30 00:05:38.697229 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:05:38.697151 2572 generic.go:356] "Generic (PLEG): container finished" podID="86def107-a402-4f54-a076-c26bcf2c0d7a" containerID="52c9ecc86b1cc337b6183dafad5e17af0b79716546de80a3b2933d9c4d2f2d29" exitCode=1 Jul 30 00:05:38.697229 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:05:38.697203 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864dbcc49f-wccmz" event={"ID":"86def107-a402-4f54-a076-c26bcf2c0d7a","Type":"ContainerDied","Data":"52c9ecc86b1cc337b6183dafad5e17af0b79716546de80a3b2933d9c4d2f2d29"} Jul 30 00:05:38.697498 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:05:38.697484 2572 scope.go:117] "RemoveContainer" containerID="52c9ecc86b1cc337b6183dafad5e17af0b79716546de80a3b2933d9c4d2f2d29" Jul 30 00:05:39.701922 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:05:39.701891 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864dbcc49f-wccmz" event={"ID":"86def107-a402-4f54-a076-c26bcf2c0d7a","Type":"ContainerStarted","Data":"68d17002523a861fd77c0cd4b82e60a5b1c67e7a058095f4b7c02704f62d777e"} Jul 30 00:05:39.702393 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:05:39.702170 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864dbcc49f-wccmz" Jul 30 00:05:39.702761 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:05:39.702746 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864dbcc49f-wccmz" Jul 30 00:05:44.861397 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:05:44.861360 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:04.736754 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:04.736721 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs\") pod \"network-metrics-daemon-nk989\" (UID: \"a38e0c1c-7133-40ff-9ba4-9a066da5c41e\") " pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:06:04.737153 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:06:04.736838 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 30 00:06:04.737153 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:06:04.736899 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs podName:a38e0c1c-7133-40ff-9ba4-9a066da5c41e nodeName:}" failed. No retries permitted until 2026-07-30 00:08:06.736885683 +0000 UTC m=+374.240143921 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs") pod "network-metrics-daemon-nk989" (UID: "a38e0c1c-7133-40ff-9ba4-9a066da5c41e") : secret "metrics-daemon-secret" not found Jul 30 00:06:15.305986 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:15.305898 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:15.406990 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:15.406956 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:18.242101 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:18.242070 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:18.752065 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:18.752027 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nx2v9"] Jul 30 00:06:18.754035 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:18.754019 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nx2v9" Jul 30 00:06:18.756603 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:18.756578 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 30 00:06:18.756738 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:18.756646 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 30 00:06:18.757780 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:18.757761 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-47ttk\"" Jul 30 00:06:18.757780 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:18.757776 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 30 00:06:18.757911 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:18.757815 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 30 00:06:18.763712 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:18.763692 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nx2v9"] Jul 30 00:06:18.778047 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:18.777995 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:18.931333 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:18.931305 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/146c3400-fc20-48bb-a456-e7b4f53f406f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nx2v9\" (UID: \"146c3400-fc20-48bb-a456-e7b4f53f406f\") " pod="openshift-insights/insights-runtime-extractor-nx2v9" Jul 30 00:06:18.931499 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:18.931344 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/146c3400-fc20-48bb-a456-e7b4f53f406f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nx2v9\" (UID: \"146c3400-fc20-48bb-a456-e7b4f53f406f\") " pod="openshift-insights/insights-runtime-extractor-nx2v9" Jul 30 00:06:18.931499 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:18.931373 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v675t\" (UniqueName: \"kubernetes.io/projected/146c3400-fc20-48bb-a456-e7b4f53f406f-kube-api-access-v675t\") pod \"insights-runtime-extractor-nx2v9\" (UID: \"146c3400-fc20-48bb-a456-e7b4f53f406f\") " pod="openshift-insights/insights-runtime-extractor-nx2v9" Jul 30 00:06:18.931499 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:18.931438 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/146c3400-fc20-48bb-a456-e7b4f53f406f-crio-socket\") pod \"insights-runtime-extractor-nx2v9\" (UID: \"146c3400-fc20-48bb-a456-e7b4f53f406f\") " pod="openshift-insights/insights-runtime-extractor-nx2v9" Jul 30 00:06:18.931623 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:18.931505 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/146c3400-fc20-48bb-a456-e7b4f53f406f-data-volume\") pod \"insights-runtime-extractor-nx2v9\" (UID: \"146c3400-fc20-48bb-a456-e7b4f53f406f\") " pod="openshift-insights/insights-runtime-extractor-nx2v9" Jul 30 00:06:19.037174 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:19.037071 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/146c3400-fc20-48bb-a456-e7b4f53f406f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nx2v9\" (UID: \"146c3400-fc20-48bb-a456-e7b4f53f406f\") " pod="openshift-insights/insights-runtime-extractor-nx2v9" Jul 30 00:06:19.037174 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:19.037121 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v675t\" (UniqueName: \"kubernetes.io/projected/146c3400-fc20-48bb-a456-e7b4f53f406f-kube-api-access-v675t\") pod \"insights-runtime-extractor-nx2v9\" (UID: \"146c3400-fc20-48bb-a456-e7b4f53f406f\") " pod="openshift-insights/insights-runtime-extractor-nx2v9" Jul 30 00:06:19.037174 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:19.037149 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/146c3400-fc20-48bb-a456-e7b4f53f406f-crio-socket\") pod \"insights-runtime-extractor-nx2v9\" (UID: \"146c3400-fc20-48bb-a456-e7b4f53f406f\") " pod="openshift-insights/insights-runtime-extractor-nx2v9" Jul 30 00:06:19.037389 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:19.037184 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/146c3400-fc20-48bb-a456-e7b4f53f406f-data-volume\") pod \"insights-runtime-extractor-nx2v9\" (UID: \"146c3400-fc20-48bb-a456-e7b4f53f406f\") " pod="openshift-insights/insights-runtime-extractor-nx2v9" Jul 30 00:06:19.037389 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:06:19.037190 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 00:06:19.037389 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:19.037217 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/146c3400-fc20-48bb-a456-e7b4f53f406f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nx2v9\" (UID: \"146c3400-fc20-48bb-a456-e7b4f53f406f\") " pod="openshift-insights/insights-runtime-extractor-nx2v9" Jul 30 00:06:19.037389 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:06:19.037277 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/146c3400-fc20-48bb-a456-e7b4f53f406f-kube-rbac-proxy-cm podName:146c3400-fc20-48bb-a456-e7b4f53f406f nodeName:}" failed. No retries permitted until 2026-07-30 00:06:19.537256685 +0000 UTC m=+267.040514938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/146c3400-fc20-48bb-a456-e7b4f53f406f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nx2v9" (UID: "146c3400-fc20-48bb-a456-e7b4f53f406f") : configmap "kube-rbac-proxy" not found Jul 30 00:06:19.037389 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:19.037305 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/146c3400-fc20-48bb-a456-e7b4f53f406f-crio-socket\") pod \"insights-runtime-extractor-nx2v9\" (UID: \"146c3400-fc20-48bb-a456-e7b4f53f406f\") " pod="openshift-insights/insights-runtime-extractor-nx2v9" Jul 30 00:06:19.037389 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:06:19.037316 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 00:06:19.037389 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:06:19.037376 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/146c3400-fc20-48bb-a456-e7b4f53f406f-insights-runtime-extractor-tls podName:146c3400-fc20-48bb-a456-e7b4f53f406f nodeName:}" failed. No retries permitted until 2026-07-30 00:06:19.537351062 +0000 UTC m=+267.040609310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/146c3400-fc20-48bb-a456-e7b4f53f406f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nx2v9" (UID: "146c3400-fc20-48bb-a456-e7b4f53f406f") : secret "insights-runtime-extractor-tls" not found Jul 30 00:06:19.037630 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:19.037530 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/146c3400-fc20-48bb-a456-e7b4f53f406f-data-volume\") pod \"insights-runtime-extractor-nx2v9\" (UID: \"146c3400-fc20-48bb-a456-e7b4f53f406f\") " pod="openshift-insights/insights-runtime-extractor-nx2v9" Jul 30 00:06:19.045514 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:19.045491 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v675t\" (UniqueName: \"kubernetes.io/projected/146c3400-fc20-48bb-a456-e7b4f53f406f-kube-api-access-v675t\") pod \"insights-runtime-extractor-nx2v9\" (UID: \"146c3400-fc20-48bb-a456-e7b4f53f406f\") " pod="openshift-insights/insights-runtime-extractor-nx2v9" Jul 30 00:06:19.540246 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:19.540199 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/146c3400-fc20-48bb-a456-e7b4f53f406f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nx2v9\" (UID: \"146c3400-fc20-48bb-a456-e7b4f53f406f\") " pod="openshift-insights/insights-runtime-extractor-nx2v9" Jul 30 00:06:19.540246 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:19.540250 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/146c3400-fc20-48bb-a456-e7b4f53f406f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nx2v9\" (UID: \"146c3400-fc20-48bb-a456-e7b4f53f406f\") " pod="openshift-insights/insights-runtime-extractor-nx2v9" Jul 30 00:06:19.540844 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:06:19.540330 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 00:06:19.540844 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:06:19.540348 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 00:06:19.540844 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:06:19.540408 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/146c3400-fc20-48bb-a456-e7b4f53f406f-insights-runtime-extractor-tls podName:146c3400-fc20-48bb-a456-e7b4f53f406f nodeName:}" failed. No retries permitted until 2026-07-30 00:06:20.540392142 +0000 UTC m=+268.043650380 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/146c3400-fc20-48bb-a456-e7b4f53f406f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nx2v9" (UID: "146c3400-fc20-48bb-a456-e7b4f53f406f") : secret "insights-runtime-extractor-tls" not found Jul 30 00:06:19.540844 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:06:19.540422 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/146c3400-fc20-48bb-a456-e7b4f53f406f-kube-rbac-proxy-cm podName:146c3400-fc20-48bb-a456-e7b4f53f406f nodeName:}" failed. No retries permitted until 2026-07-30 00:06:20.540415531 +0000 UTC m=+268.043673769 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/146c3400-fc20-48bb-a456-e7b4f53f406f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nx2v9" (UID: "146c3400-fc20-48bb-a456-e7b4f53f406f") : configmap "kube-rbac-proxy" not found Jul 30 00:06:19.592350 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:19.592318 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:19.692726 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:19.692663 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:20.545313 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:20.545282 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/146c3400-fc20-48bb-a456-e7b4f53f406f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nx2v9\" (UID: \"146c3400-fc20-48bb-a456-e7b4f53f406f\") " pod="openshift-insights/insights-runtime-extractor-nx2v9" Jul 30 00:06:20.545678 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:20.545352 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/146c3400-fc20-48bb-a456-e7b4f53f406f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nx2v9\" (UID: \"146c3400-fc20-48bb-a456-e7b4f53f406f\") " pod="openshift-insights/insights-runtime-extractor-nx2v9" Jul 30 00:06:20.545678 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:06:20.545402 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 00:06:20.545678 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:06:20.545432 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 00:06:20.545678 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:06:20.545473 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/146c3400-fc20-48bb-a456-e7b4f53f406f-kube-rbac-proxy-cm podName:146c3400-fc20-48bb-a456-e7b4f53f406f nodeName:}" failed. No retries permitted until 2026-07-30 00:06:22.545453907 +0000 UTC m=+270.048712145 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/146c3400-fc20-48bb-a456-e7b4f53f406f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nx2v9" (UID: "146c3400-fc20-48bb-a456-e7b4f53f406f") : configmap "kube-rbac-proxy" not found Jul 30 00:06:20.545678 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:06:20.545489 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/146c3400-fc20-48bb-a456-e7b4f53f406f-insights-runtime-extractor-tls podName:146c3400-fc20-48bb-a456-e7b4f53f406f nodeName:}" failed. No retries permitted until 2026-07-30 00:06:22.545482699 +0000 UTC m=+270.048740937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/146c3400-fc20-48bb-a456-e7b4f53f406f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nx2v9" (UID: "146c3400-fc20-48bb-a456-e7b4f53f406f") : secret "insights-runtime-extractor-tls" not found Jul 30 00:06:20.931098 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:20.931072 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zsmpb_6bf193c2-d465-4e6c-abaf-afc0f08815fa/dns-node-resolver/0.log" Jul 30 00:06:21.274506 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:21.274428 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:22.132369 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:22.132336 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lbgfr_aa41535c-676c-430a-9151-7257622c4076/node-ca/0.log" Jul 30 00:06:22.559727 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:22.559638 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/146c3400-fc20-48bb-a456-e7b4f53f406f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nx2v9\" (UID: \"146c3400-fc20-48bb-a456-e7b4f53f406f\") " pod="openshift-insights/insights-runtime-extractor-nx2v9" Jul 30 00:06:22.559727 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:22.559681 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/146c3400-fc20-48bb-a456-e7b4f53f406f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nx2v9\" (UID: \"146c3400-fc20-48bb-a456-e7b4f53f406f\") " pod="openshift-insights/insights-runtime-extractor-nx2v9" Jul 30 00:06:22.559943 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:06:22.559764 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 00:06:22.559943 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:06:22.559786 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 00:06:22.559943 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:06:22.559827 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/146c3400-fc20-48bb-a456-e7b4f53f406f-kube-rbac-proxy-cm podName:146c3400-fc20-48bb-a456-e7b4f53f406f nodeName:}" failed. No retries permitted until 2026-07-30 00:06:26.559809567 +0000 UTC m=+274.063067805 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/146c3400-fc20-48bb-a456-e7b4f53f406f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nx2v9" (UID: "146c3400-fc20-48bb-a456-e7b4f53f406f") : configmap "kube-rbac-proxy" not found Jul 30 00:06:22.559943 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:06:22.559845 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/146c3400-fc20-48bb-a456-e7b4f53f406f-insights-runtime-extractor-tls podName:146c3400-fc20-48bb-a456-e7b4f53f406f nodeName:}" failed. No retries permitted until 2026-07-30 00:06:26.55983699 +0000 UTC m=+274.063095235 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/146c3400-fc20-48bb-a456-e7b4f53f406f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nx2v9" (UID: "146c3400-fc20-48bb-a456-e7b4f53f406f") : secret "insights-runtime-extractor-tls" not found Jul 30 00:06:22.732648 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:22.732609 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mksd5_b9b8edba-eb6a-4906-97a4-4eb273367aed/serve-healthcheck-canary/0.log" Jul 30 00:06:22.732813 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:22.732720 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:25.000040 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:25.000010 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:25.109473 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:25.109443 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:25.205194 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:25.205163 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:25.305620 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:25.305550 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:26.225720 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:26.225685 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:26.421505 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:26.421471 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:26.521271 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:26.521196 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:26.585359 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:26.585331 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/146c3400-fc20-48bb-a456-e7b4f53f406f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nx2v9\" (UID: \"146c3400-fc20-48bb-a456-e7b4f53f406f\") " pod="openshift-insights/insights-runtime-extractor-nx2v9" Jul 30 00:06:26.585464 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:26.585363 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/146c3400-fc20-48bb-a456-e7b4f53f406f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nx2v9\" (UID: \"146c3400-fc20-48bb-a456-e7b4f53f406f\") " pod="openshift-insights/insights-runtime-extractor-nx2v9" Jul 30 00:06:26.585507 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:06:26.585461 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 00:06:26.585540 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:06:26.585520 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/146c3400-fc20-48bb-a456-e7b4f53f406f-insights-runtime-extractor-tls podName:146c3400-fc20-48bb-a456-e7b4f53f406f nodeName:}" failed. No retries permitted until 2026-07-30 00:06:34.585503986 +0000 UTC m=+282.088762223 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/146c3400-fc20-48bb-a456-e7b4f53f406f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nx2v9" (UID: "146c3400-fc20-48bb-a456-e7b4f53f406f") : secret "insights-runtime-extractor-tls" not found Jul 30 00:06:26.585838 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:26.585822 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/146c3400-fc20-48bb-a456-e7b4f53f406f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nx2v9\" (UID: \"146c3400-fc20-48bb-a456-e7b4f53f406f\") " pod="openshift-insights/insights-runtime-extractor-nx2v9" Jul 30 00:06:27.096564 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:27.096537 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:27.304873 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:27.304841 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:27.697071 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:27.697041 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:27.896398 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:27.896365 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:28.301747 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:28.301708 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:28.401680 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:28.401644 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:28.894310 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:28.894281 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:29.013725 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:29.013691 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:29.299539 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:29.299460 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:29.347828 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:29.347791 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:29.495875 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:29.495846 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:29.897012 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:29.896977 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:30.497922 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:30.497893 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:30.912880 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:30.912849 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:31.297580 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:31.297505 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:31.497209 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:31.497180 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:31.546952 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:31.546928 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:31.897699 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:31.897672 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:32.703492 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:32.703459 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:33.097445 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:33.097416 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:33.308179 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:33.308151 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:33.500967 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:33.500886 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:33.509976 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:06:33.509952 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-tb6c5" podUID="34dd2a14-f850-4a05-ab6b-9c0c0196ed0a" Jul 30 00:06:33.698146 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:33.698098 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:33.849815 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:33.849788 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tb6c5" Jul 30 00:06:34.639925 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:34.639849 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/146c3400-fc20-48bb-a456-e7b4f53f406f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nx2v9\" (UID: \"146c3400-fc20-48bb-a456-e7b4f53f406f\") " pod="openshift-insights/insights-runtime-extractor-nx2v9" Jul 30 00:06:34.640080 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:06:34.639997 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 00:06:34.640080 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:06:34.640063 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/146c3400-fc20-48bb-a456-e7b4f53f406f-insights-runtime-extractor-tls podName:146c3400-fc20-48bb-a456-e7b4f53f406f nodeName:}" failed. No retries permitted until 2026-07-30 00:06:50.640048299 +0000 UTC m=+298.143306537 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/146c3400-fc20-48bb-a456-e7b4f53f406f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nx2v9" (UID: "146c3400-fc20-48bb-a456-e7b4f53f406f") : secret "insights-runtime-extractor-tls" not found Jul 30 00:06:34.893229 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:34.893150 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:35.217373 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:35.217297 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:35.318371 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:35.318342 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:35.495457 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:35.495383 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:35.699171 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:35.699144 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:36.386662 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:36.386630 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:37.352906 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:37.352870 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:37.356586 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:37.356569 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls\") pod \"dns-default-tb6c5\" (UID: \"34dd2a14-f850-4a05-ab6b-9c0c0196ed0a\") " pod="openshift-dns/dns-default-tb6c5" Jul 30 00:06:37.358738 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:37.358715 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34dd2a14-f850-4a05-ab6b-9c0c0196ed0a-metrics-tls\") pod \"dns-default-tb6c5\" (UID: \"34dd2a14-f850-4a05-ab6b-9c0c0196ed0a\") " pod="openshift-dns/dns-default-tb6c5" Jul 30 00:06:37.453643 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:37.453616 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nxhcc\"" Jul 30 00:06:37.461884 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:37.461861 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tb6c5" Jul 30 00:06:37.573276 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:37.573246 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tb6c5"] Jul 30 00:06:37.576933 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:06:37.576907 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34dd2a14_f850_4a05_ab6b_9c0c0196ed0a.slice/crio-622c181616e4a497e3319ed463e60ddb2c040c6186050b9e8f430f31415d7242 WatchSource:0}: Error finding container 622c181616e4a497e3319ed463e60ddb2c040c6186050b9e8f430f31415d7242: Status 404 returned error can't find the container with id 622c181616e4a497e3319ed463e60ddb2c040c6186050b9e8f430f31415d7242 Jul 30 00:06:37.862513 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:37.862484 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tb6c5" event={"ID":"34dd2a14-f850-4a05-ab6b-9c0c0196ed0a","Type":"ContainerStarted","Data":"622c181616e4a497e3319ed463e60ddb2c040c6186050b9e8f430f31415d7242"} Jul 30 00:06:39.819605 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.819567 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65995cc6bf-9kq6s"] Jul 30 00:06:39.821551 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.821533 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" Jul 30 00:06:39.826059 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.826038 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 30 00:06:39.826176 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.826086 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 30 00:06:39.826331 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.826300 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pt2ls\"" Jul 30 00:06:39.827535 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.827515 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 30 00:06:39.831759 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.831737 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 30 00:06:39.834761 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.834740 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65995cc6bf-9kq6s"] Jul 30 00:06:39.872497 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.872471 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a96f39c-93c9-4cc7-b2c2-e189529a15f4-image-registry-private-configuration\") pod \"image-registry-65995cc6bf-9kq6s\" (UID: \"0a96f39c-93c9-4cc7-b2c2-e189529a15f4\") " pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" Jul 30 00:06:39.872620 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.872511 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a96f39c-93c9-4cc7-b2c2-e189529a15f4-ca-trust-extracted\") pod \"image-registry-65995cc6bf-9kq6s\" (UID: \"0a96f39c-93c9-4cc7-b2c2-e189529a15f4\") " pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" Jul 30 00:06:39.872620 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.872536 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtl2n\" (UniqueName: \"kubernetes.io/projected/0a96f39c-93c9-4cc7-b2c2-e189529a15f4-kube-api-access-wtl2n\") pod \"image-registry-65995cc6bf-9kq6s\" (UID: \"0a96f39c-93c9-4cc7-b2c2-e189529a15f4\") " pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" Jul 30 00:06:39.872747 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.872632 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a96f39c-93c9-4cc7-b2c2-e189529a15f4-registry-certificates\") pod \"image-registry-65995cc6bf-9kq6s\" (UID: \"0a96f39c-93c9-4cc7-b2c2-e189529a15f4\") " pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" Jul 30 00:06:39.872747 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.872671 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a96f39c-93c9-4cc7-b2c2-e189529a15f4-bound-sa-token\") pod \"image-registry-65995cc6bf-9kq6s\" (UID: \"0a96f39c-93c9-4cc7-b2c2-e189529a15f4\") " pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" Jul 30 00:06:39.872747 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.872706 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a96f39c-93c9-4cc7-b2c2-e189529a15f4-trusted-ca\") pod \"image-registry-65995cc6bf-9kq6s\" (UID: \"0a96f39c-93c9-4cc7-b2c2-e189529a15f4\") " pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" Jul 30 00:06:39.872747 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.872737 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a96f39c-93c9-4cc7-b2c2-e189529a15f4-registry-tls\") pod \"image-registry-65995cc6bf-9kq6s\" (UID: \"0a96f39c-93c9-4cc7-b2c2-e189529a15f4\") " pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" Jul 30 00:06:39.872883 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.872793 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a96f39c-93c9-4cc7-b2c2-e189529a15f4-installation-pull-secrets\") pod \"image-registry-65995cc6bf-9kq6s\" (UID: \"0a96f39c-93c9-4cc7-b2c2-e189529a15f4\") " pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" Jul 30 00:06:39.880026 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.880002 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:39.973896 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.973861 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a96f39c-93c9-4cc7-b2c2-e189529a15f4-image-registry-private-configuration\") pod \"image-registry-65995cc6bf-9kq6s\" (UID: \"0a96f39c-93c9-4cc7-b2c2-e189529a15f4\") " pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" Jul 30 00:06:39.974059 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.973919 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a96f39c-93c9-4cc7-b2c2-e189529a15f4-ca-trust-extracted\") pod \"image-registry-65995cc6bf-9kq6s\" (UID: \"0a96f39c-93c9-4cc7-b2c2-e189529a15f4\") " pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" Jul 30 00:06:39.974059 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.973950 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtl2n\" (UniqueName: \"kubernetes.io/projected/0a96f39c-93c9-4cc7-b2c2-e189529a15f4-kube-api-access-wtl2n\") pod \"image-registry-65995cc6bf-9kq6s\" (UID: \"0a96f39c-93c9-4cc7-b2c2-e189529a15f4\") " pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" Jul 30 00:06:39.974059 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.973977 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a96f39c-93c9-4cc7-b2c2-e189529a15f4-registry-certificates\") pod \"image-registry-65995cc6bf-9kq6s\" (UID: \"0a96f39c-93c9-4cc7-b2c2-e189529a15f4\") " pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" Jul 30 00:06:39.974059 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.974000 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a96f39c-93c9-4cc7-b2c2-e189529a15f4-bound-sa-token\") pod \"image-registry-65995cc6bf-9kq6s\" (UID: \"0a96f39c-93c9-4cc7-b2c2-e189529a15f4\") " pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" Jul 30 00:06:39.974059 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.974035 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a96f39c-93c9-4cc7-b2c2-e189529a15f4-trusted-ca\") pod \"image-registry-65995cc6bf-9kq6s\" (UID: \"0a96f39c-93c9-4cc7-b2c2-e189529a15f4\") " pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" Jul 30 00:06:39.974311 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.974064 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a96f39c-93c9-4cc7-b2c2-e189529a15f4-registry-tls\") pod \"image-registry-65995cc6bf-9kq6s\" (UID: \"0a96f39c-93c9-4cc7-b2c2-e189529a15f4\") " pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" Jul 30 00:06:39.974311 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.974121 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a96f39c-93c9-4cc7-b2c2-e189529a15f4-installation-pull-secrets\") pod \"image-registry-65995cc6bf-9kq6s\" (UID: \"0a96f39c-93c9-4cc7-b2c2-e189529a15f4\") " pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" Jul 30 00:06:39.974669 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.974642 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a96f39c-93c9-4cc7-b2c2-e189529a15f4-ca-trust-extracted\") pod \"image-registry-65995cc6bf-9kq6s\" (UID: \"0a96f39c-93c9-4cc7-b2c2-e189529a15f4\") " pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" Jul 30 00:06:39.975391 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.975366 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a96f39c-93c9-4cc7-b2c2-e189529a15f4-trusted-ca\") pod \"image-registry-65995cc6bf-9kq6s\" (UID: \"0a96f39c-93c9-4cc7-b2c2-e189529a15f4\") " pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" Jul 30 00:06:39.975495 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.975413 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a96f39c-93c9-4cc7-b2c2-e189529a15f4-registry-certificates\") pod \"image-registry-65995cc6bf-9kq6s\" (UID: \"0a96f39c-93c9-4cc7-b2c2-e189529a15f4\") " pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" Jul 30 00:06:39.976780 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.976757 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a96f39c-93c9-4cc7-b2c2-e189529a15f4-image-registry-private-configuration\") pod \"image-registry-65995cc6bf-9kq6s\" (UID: \"0a96f39c-93c9-4cc7-b2c2-e189529a15f4\") " pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" Jul 30 00:06:39.977034 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.977012 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a96f39c-93c9-4cc7-b2c2-e189529a15f4-installation-pull-secrets\") pod \"image-registry-65995cc6bf-9kq6s\" (UID: \"0a96f39c-93c9-4cc7-b2c2-e189529a15f4\") " pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" Jul 30 00:06:39.977308 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.977286 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a96f39c-93c9-4cc7-b2c2-e189529a15f4-registry-tls\") pod \"image-registry-65995cc6bf-9kq6s\" (UID: \"0a96f39c-93c9-4cc7-b2c2-e189529a15f4\") " pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" Jul 30 00:06:39.982219 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.982192 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtl2n\" (UniqueName: \"kubernetes.io/projected/0a96f39c-93c9-4cc7-b2c2-e189529a15f4-kube-api-access-wtl2n\") pod \"image-registry-65995cc6bf-9kq6s\" (UID: \"0a96f39c-93c9-4cc7-b2c2-e189529a15f4\") " pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" Jul 30 00:06:39.982305 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.982291 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a96f39c-93c9-4cc7-b2c2-e189529a15f4-bound-sa-token\") pod \"image-registry-65995cc6bf-9kq6s\" (UID: \"0a96f39c-93c9-4cc7-b2c2-e189529a15f4\") " pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" Jul 30 00:06:39.984519 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:39.984456 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:40.092888 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:40.092866 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:40.136580 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:40.136558 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" Jul 30 00:06:40.194951 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:40.190320 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:40.275271 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:40.275239 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65995cc6bf-9kq6s"] Jul 30 00:06:40.278088 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:40.278014 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:40.278265 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:06:40.278244 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a96f39c_93c9_4cc7_b2c2_e189529a15f4.slice/crio-766b3bf750724e3b37f8aba9acdce3f14d9bc4203b1366cef6fcff528eb7521e WatchSource:0}: Error finding container 766b3bf750724e3b37f8aba9acdce3f14d9bc4203b1366cef6fcff528eb7521e: Status 404 returned error can't find the container with id 766b3bf750724e3b37f8aba9acdce3f14d9bc4203b1366cef6fcff528eb7521e Jul 30 00:06:40.331924 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:40.331879 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:40.432491 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:40.432420 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:40.556193 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:40.556153 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:40.609389 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:40.609359 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:40.750084 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:40.750010 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:40.873397 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:40.873358 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tb6c5" event={"ID":"34dd2a14-f850-4a05-ab6b-9c0c0196ed0a","Type":"ContainerStarted","Data":"37eb93db1844b8487bc6e66bd6f4d389da68a6c4a362190d43cc676366154378"} Jul 30 00:06:40.873792 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:40.873420 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tb6c5" event={"ID":"34dd2a14-f850-4a05-ab6b-9c0c0196ed0a","Type":"ContainerStarted","Data":"fa894f59be799e078a98b996fe3f19d8430483b07a0db1b62f9d99ad2813cc98"} Jul 30 00:06:40.873792 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:40.873447 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tb6c5" Jul 30 00:06:40.874857 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:40.874833 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" event={"ID":"0a96f39c-93c9-4cc7-b2c2-e189529a15f4","Type":"ContainerStarted","Data":"3e0c1b98686b7559e3d42ed60e6a7865c117bb310c4771be82b6626bd4691f21"} Jul 30 00:06:40.874957 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:40.874874 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" event={"ID":"0a96f39c-93c9-4cc7-b2c2-e189529a15f4","Type":"ContainerStarted","Data":"766b3bf750724e3b37f8aba9acdce3f14d9bc4203b1366cef6fcff528eb7521e"} Jul 30 00:06:40.874957 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:40.874937 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" Jul 30 00:06:40.890398 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:40.890351 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tb6c5" podStartSLOduration=251.381595786 podStartE2EDuration="4m13.890335566s" totalImagesPullingTime="2.50873978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:06:40.888697335 +0000 UTC m=+288.391955596" watchObservedRunningTime="2026-07-30 00:06:40.890335566 +0000 UTC m=+288.393593825" Jul 30 00:06:40.906680 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:40.906642 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" podStartSLOduration=1.906616826 podStartE2EDuration="1.906616826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:06:40.905503909 +0000 UTC m=+288.408762171" watchObservedRunningTime="2026-07-30 00:06:40.906616826 +0000 UTC m=+288.409875084" Jul 30 00:06:41.701753 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:41.701720 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:41.909465 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:41.909437 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:42.168186 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:42.168157 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:42.227779 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:42.227749 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:48.987643 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:06:48.987603 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-nk989" podUID="a38e0c1c-7133-40ff-9ba4-9a066da5c41e" Jul 30 00:06:48.988335 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:48.988314 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:49.088116 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:49.088088 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:49.850047 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:49.849998 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:49.950691 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:49.950663 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:50.007799 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:50.007772 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:50.649164 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:50.649130 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/146c3400-fc20-48bb-a456-e7b4f53f406f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nx2v9\" (UID: \"146c3400-fc20-48bb-a456-e7b4f53f406f\") " pod="openshift-insights/insights-runtime-extractor-nx2v9" Jul 30 00:06:50.651356 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:50.651323 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/146c3400-fc20-48bb-a456-e7b4f53f406f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nx2v9\" (UID: \"146c3400-fc20-48bb-a456-e7b4f53f406f\") " pod="openshift-insights/insights-runtime-extractor-nx2v9" Jul 30 00:06:50.863003 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:50.862968 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nx2v9" Jul 30 00:06:50.880566 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:50.880544 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tb6c5" Jul 30 00:06:50.991234 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:50.991209 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nx2v9"] Jul 30 00:06:50.994056 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:06:50.994029 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146c3400_fc20_48bb_a456_e7b4f53f406f.slice/crio-7cda286453eda498f8a11d2bd6f4c8d5b97537a80f67ac7260f91b4104ca37fc WatchSource:0}: Error finding container 7cda286453eda498f8a11d2bd6f4c8d5b97537a80f67ac7260f91b4104ca37fc: Status 404 returned error can't find the container with id 7cda286453eda498f8a11d2bd6f4c8d5b97537a80f67ac7260f91b4104ca37fc Jul 30 00:06:51.259998 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:51.259915 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:51.909870 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:51.909841 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nx2v9" event={"ID":"146c3400-fc20-48bb-a456-e7b4f53f406f","Type":"ContainerStarted","Data":"7cda286453eda498f8a11d2bd6f4c8d5b97537a80f67ac7260f91b4104ca37fc"} Jul 30 00:06:52.441034 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:52.441003 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:52.497654 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:52.497627 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:52.875716 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:52.875681 2572 kubelet.go:1648] "Image garbage collection succeeded" Jul 30 00:06:53.041090 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:53.040935 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:53.075751 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:53.075724 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-lmwz5"] Jul 30 00:06:53.078439 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:53.078419 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-lmwz5" Jul 30 00:06:53.091807 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:53.091782 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 30 00:06:53.093313 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:53.093132 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-dx9v9\"" Jul 30 00:06:53.093313 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:53.093185 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 30 00:06:53.093313 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:53.093199 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 30 00:06:53.093313 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:53.093241 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 30 00:06:53.093313 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:53.093190 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 30 00:06:53.102496 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:53.102466 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-lmwz5"] Jul 30 00:06:53.133061 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:53.133005 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:53.168080 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:53.168047 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/63ffb722-7c47-41f7-8a27-96664c33e1e5-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-lmwz5\" (UID: \"63ffb722-7c47-41f7-8a27-96664c33e1e5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lmwz5" Jul 30 00:06:53.168218 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:53.168121 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63ffb722-7c47-41f7-8a27-96664c33e1e5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-lmwz5\" (UID: \"63ffb722-7c47-41f7-8a27-96664c33e1e5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lmwz5" Jul 30 00:06:53.168218 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:53.168192 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63ffb722-7c47-41f7-8a27-96664c33e1e5-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-lmwz5\" (UID: \"63ffb722-7c47-41f7-8a27-96664c33e1e5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lmwz5" Jul 30 00:06:53.168315 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:53.168296 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sqh4\" (UniqueName: \"kubernetes.io/projected/63ffb722-7c47-41f7-8a27-96664c33e1e5-kube-api-access-7sqh4\") pod \"prometheus-operator-84d69f794d-lmwz5\" (UID: \"63ffb722-7c47-41f7-8a27-96664c33e1e5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lmwz5" Jul 30 00:06:53.194161 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:53.194135 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:53.269389 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:53.269360 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/63ffb722-7c47-41f7-8a27-96664c33e1e5-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-lmwz5\" (UID: \"63ffb722-7c47-41f7-8a27-96664c33e1e5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lmwz5" Jul 30 00:06:53.269514 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:53.269407 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63ffb722-7c47-41f7-8a27-96664c33e1e5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-lmwz5\" (UID: \"63ffb722-7c47-41f7-8a27-96664c33e1e5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lmwz5" Jul 30 00:06:53.269514 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:53.269443 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63ffb722-7c47-41f7-8a27-96664c33e1e5-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-lmwz5\" (UID: \"63ffb722-7c47-41f7-8a27-96664c33e1e5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lmwz5" Jul 30 00:06:53.269619 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:53.269528 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sqh4\" (UniqueName: \"kubernetes.io/projected/63ffb722-7c47-41f7-8a27-96664c33e1e5-kube-api-access-7sqh4\") pod \"prometheus-operator-84d69f794d-lmwz5\" (UID: \"63ffb722-7c47-41f7-8a27-96664c33e1e5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lmwz5" Jul 30 00:06:53.270425 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:53.270405 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63ffb722-7c47-41f7-8a27-96664c33e1e5-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-lmwz5\" (UID: \"63ffb722-7c47-41f7-8a27-96664c33e1e5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lmwz5" Jul 30 00:06:53.272216 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:53.272192 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/63ffb722-7c47-41f7-8a27-96664c33e1e5-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-lmwz5\" (UID: \"63ffb722-7c47-41f7-8a27-96664c33e1e5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lmwz5" Jul 30 00:06:53.272321 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:53.272237 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63ffb722-7c47-41f7-8a27-96664c33e1e5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-lmwz5\" (UID: \"63ffb722-7c47-41f7-8a27-96664c33e1e5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lmwz5" Jul 30 00:06:53.277248 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:53.277224 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sqh4\" (UniqueName: \"kubernetes.io/projected/63ffb722-7c47-41f7-8a27-96664c33e1e5-kube-api-access-7sqh4\") pod \"prometheus-operator-84d69f794d-lmwz5\" (UID: \"63ffb722-7c47-41f7-8a27-96664c33e1e5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-lmwz5" Jul 30 00:06:53.388289 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:53.388214 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-lmwz5" Jul 30 00:06:53.527367 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:53.527343 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-lmwz5"] Jul 30 00:06:53.529817 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:06:53.529785 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ffb722_7c47_41f7_8a27_96664c33e1e5.slice/crio-d90164310513a18223a1db08e9fced5c02843cc9a3c8a4d265c1753757804cb5 WatchSource:0}: Error finding container d90164310513a18223a1db08e9fced5c02843cc9a3c8a4d265c1753757804cb5: Status 404 returned error can't find the container with id d90164310513a18223a1db08e9fced5c02843cc9a3c8a4d265c1753757804cb5 Jul 30 00:06:53.917678 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:53.917643 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-lmwz5" event={"ID":"63ffb722-7c47-41f7-8a27-96664c33e1e5","Type":"ContainerStarted","Data":"d90164310513a18223a1db08e9fced5c02843cc9a3c8a4d265c1753757804cb5"} Jul 30 00:06:57.138205 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:57.137752 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 00:06:57.274128 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:57.274038 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:57.765967 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:57.765938 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:57.930839 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:57.930810 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nx2v9" event={"ID":"146c3400-fc20-48bb-a456-e7b4f53f406f","Type":"ContainerStarted","Data":"0424c3a13f64d5dba5ca365f4b5fc43135ddd2f91df232da44c83c9bae8ad174"} Jul 30 00:06:57.932479 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:57.932454 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-lmwz5" event={"ID":"63ffb722-7c47-41f7-8a27-96664c33e1e5","Type":"ContainerStarted","Data":"2ba4cce5006f5ce9b9d56bda2a9a5426fd44af6cd8bb3a1c8d6511f2f32bf029"} Jul 30 00:06:57.932601 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:57.932501 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-lmwz5" event={"ID":"63ffb722-7c47-41f7-8a27-96664c33e1e5","Type":"ContainerStarted","Data":"75da06cf756a4bb8b37f532fdd204e45bceea3a1e30014c23deba5a9078ad376"} Jul 30 00:06:57.947685 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:57.947648 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-lmwz5" podStartSLOduration=1.4558554670000001 podStartE2EDuration="4.947636721s" totalImagesPullingTime="3.491781254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:06:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:06:57.947632759 +0000 UTC m=+305.450891018" watchObservedRunningTime="2026-07-30 00:06:57.947636721 +0000 UTC m=+305.450894976" Jul 30 00:06:58.181362 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:58.181336 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:58.350954 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:58.350921 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:58.451446 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:58.451376 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:58.500320 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:58.500296 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:58.583336 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:58.583309 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:59.171321 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.171279 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:59.378917 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.378844 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:59.428410 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.428338 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8h6k9"] Jul 30 00:06:59.431495 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.431470 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.434142 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.434095 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zmzzb\"" Jul 30 00:06:59.434302 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.434095 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 30 00:06:59.434593 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.434405 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 30 00:06:59.434593 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.434470 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 30 00:06:59.462429 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.462399 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:59.517258 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.517231 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1c121fea-270e-4763-aff7-53348028373d-node-exporter-textfile\") pod \"node-exporter-8h6k9\" (UID: \"1c121fea-270e-4763-aff7-53348028373d\") " pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.517385 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.517260 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1c121fea-270e-4763-aff7-53348028373d-node-exporter-tls\") pod \"node-exporter-8h6k9\" (UID: \"1c121fea-270e-4763-aff7-53348028373d\") " pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.517385 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.517279 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f62dm\" (UniqueName: \"kubernetes.io/projected/1c121fea-270e-4763-aff7-53348028373d-kube-api-access-f62dm\") pod \"node-exporter-8h6k9\" (UID: \"1c121fea-270e-4763-aff7-53348028373d\") " pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.517385 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.517310 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1c121fea-270e-4763-aff7-53348028373d-node-exporter-wtmp\") pod \"node-exporter-8h6k9\" (UID: \"1c121fea-270e-4763-aff7-53348028373d\") " pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.517525 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.517395 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1c121fea-270e-4763-aff7-53348028373d-node-exporter-accelerators-collector-config\") pod \"node-exporter-8h6k9\" (UID: \"1c121fea-270e-4763-aff7-53348028373d\") " pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.517525 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.517446 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c121fea-270e-4763-aff7-53348028373d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8h6k9\" (UID: \"1c121fea-270e-4763-aff7-53348028373d\") " pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.517525 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.517472 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c121fea-270e-4763-aff7-53348028373d-sys\") pod \"node-exporter-8h6k9\" (UID: \"1c121fea-270e-4763-aff7-53348028373d\") " pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.517525 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.517487 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c121fea-270e-4763-aff7-53348028373d-metrics-client-ca\") pod \"node-exporter-8h6k9\" (UID: \"1c121fea-270e-4763-aff7-53348028373d\") " pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.517525 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.517507 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1c121fea-270e-4763-aff7-53348028373d-root\") pod \"node-exporter-8h6k9\" (UID: \"1c121fea-270e-4763-aff7-53348028373d\") " pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.608633 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.608597 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:59.618317 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.618293 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c121fea-270e-4763-aff7-53348028373d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8h6k9\" (UID: \"1c121fea-270e-4763-aff7-53348028373d\") " pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.618446 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.618325 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c121fea-270e-4763-aff7-53348028373d-sys\") pod \"node-exporter-8h6k9\" (UID: \"1c121fea-270e-4763-aff7-53348028373d\") " pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.618446 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.618342 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c121fea-270e-4763-aff7-53348028373d-metrics-client-ca\") pod \"node-exporter-8h6k9\" (UID: \"1c121fea-270e-4763-aff7-53348028373d\") " pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.618446 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.618362 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1c121fea-270e-4763-aff7-53348028373d-root\") pod \"node-exporter-8h6k9\" (UID: \"1c121fea-270e-4763-aff7-53348028373d\") " pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.618446 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.618411 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1c121fea-270e-4763-aff7-53348028373d-node-exporter-textfile\") pod \"node-exporter-8h6k9\" (UID: \"1c121fea-270e-4763-aff7-53348028373d\") " pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.618643 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.618448 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1c121fea-270e-4763-aff7-53348028373d-node-exporter-tls\") pod \"node-exporter-8h6k9\" (UID: \"1c121fea-270e-4763-aff7-53348028373d\") " pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.618643 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.618476 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f62dm\" (UniqueName: \"kubernetes.io/projected/1c121fea-270e-4763-aff7-53348028373d-kube-api-access-f62dm\") pod \"node-exporter-8h6k9\" (UID: \"1c121fea-270e-4763-aff7-53348028373d\") " pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.618643 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.618491 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c121fea-270e-4763-aff7-53348028373d-sys\") pod \"node-exporter-8h6k9\" (UID: \"1c121fea-270e-4763-aff7-53348028373d\") " pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.618643 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.618541 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1c121fea-270e-4763-aff7-53348028373d-node-exporter-wtmp\") pod \"node-exporter-8h6k9\" (UID: \"1c121fea-270e-4763-aff7-53348028373d\") " pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.618643 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.618608 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1c121fea-270e-4763-aff7-53348028373d-root\") pod \"node-exporter-8h6k9\" (UID: \"1c121fea-270e-4763-aff7-53348028373d\") " pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.618856 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.618644 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1c121fea-270e-4763-aff7-53348028373d-node-exporter-accelerators-collector-config\") pod \"node-exporter-8h6k9\" (UID: \"1c121fea-270e-4763-aff7-53348028373d\") " pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.618856 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.618692 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1c121fea-270e-4763-aff7-53348028373d-node-exporter-wtmp\") pod \"node-exporter-8h6k9\" (UID: \"1c121fea-270e-4763-aff7-53348028373d\") " pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.618856 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.618781 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1c121fea-270e-4763-aff7-53348028373d-node-exporter-textfile\") pod \"node-exporter-8h6k9\" (UID: \"1c121fea-270e-4763-aff7-53348028373d\") " pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.619011 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.618909 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c121fea-270e-4763-aff7-53348028373d-metrics-client-ca\") pod \"node-exporter-8h6k9\" (UID: \"1c121fea-270e-4763-aff7-53348028373d\") " pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.619118 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.619089 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1c121fea-270e-4763-aff7-53348028373d-node-exporter-accelerators-collector-config\") pod \"node-exporter-8h6k9\" (UID: \"1c121fea-270e-4763-aff7-53348028373d\") " pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.620637 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.620617 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c121fea-270e-4763-aff7-53348028373d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8h6k9\" (UID: \"1c121fea-270e-4763-aff7-53348028373d\") " pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.620848 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.620828 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1c121fea-270e-4763-aff7-53348028373d-node-exporter-tls\") pod \"node-exporter-8h6k9\" (UID: \"1c121fea-270e-4763-aff7-53348028373d\") " pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.629923 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.629899 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f62dm\" (UniqueName: \"kubernetes.io/projected/1c121fea-270e-4763-aff7-53348028373d-kube-api-access-f62dm\") pod \"node-exporter-8h6k9\" (UID: \"1c121fea-270e-4763-aff7-53348028373d\") " pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.709764 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.709682 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:59.742713 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.742687 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8h6k9" Jul 30 00:06:59.809737 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.809702 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:06:59.992221 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:06:59.992147 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:00.371265 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:00.371232 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:00.467331 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:00.467298 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:00.568065 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:00.568028 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:00.926322 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:00.926294 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:01.527476 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:07:01.527446 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c121fea_270e_4763_aff7_53348028373d.slice/crio-84487bc8ebc56466ea4281f414a75193290debe2c5b39e53ae0254adfda1f6c2 WatchSource:0}: Error finding container 84487bc8ebc56466ea4281f414a75193290debe2c5b39e53ae0254adfda1f6c2: Status 404 returned error can't find the container with id 84487bc8ebc56466ea4281f414a75193290debe2c5b39e53ae0254adfda1f6c2 Jul 30 00:07:01.797997 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:01.797932 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:01.881955 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:01.881928 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65995cc6bf-9kq6s" Jul 30 00:07:01.901915 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:01.901887 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:01.950820 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:01.950787 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8h6k9" event={"ID":"1c121fea-270e-4763-aff7-53348028373d","Type":"ContainerStarted","Data":"84487bc8ebc56466ea4281f414a75193290debe2c5b39e53ae0254adfda1f6c2"} Jul 30 00:07:01.952967 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:01.952936 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nx2v9" event={"ID":"146c3400-fc20-48bb-a456-e7b4f53f406f","Type":"ContainerStarted","Data":"3a49f9f39cd28b776f6b644cbc776d1cad46044a7f70c57c083082cd00cadfd7"} Jul 30 00:07:02.003262 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:02.003228 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:02.215344 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:02.215309 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:02.318550 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:02.318516 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:02.416906 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:02.416877 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:02.669394 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:02.669353 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:02.768742 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:02.768586 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:02.958289 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:02.958242 2572 generic.go:356] "Generic (PLEG): container finished" podID="1c121fea-270e-4763-aff7-53348028373d" containerID="c618a6798f944b5629b1c336139a3470ac3b29f4680b3121b311750647f7721e" exitCode=0 Jul 30 00:07:02.958451 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:02.958306 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8h6k9" event={"ID":"1c121fea-270e-4763-aff7-53348028373d","Type":"ContainerDied","Data":"c618a6798f944b5629b1c336139a3470ac3b29f4680b3121b311750647f7721e"} Jul 30 00:07:03.212703 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:03.212671 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:03.964849 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:03.964804 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8h6k9" event={"ID":"1c121fea-270e-4763-aff7-53348028373d","Type":"ContainerStarted","Data":"f3bacab084d4ad8643559227f9fa82a0d8a44ead1c1845fe76b46aacc1de784c"} Jul 30 00:07:03.965276 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:03.964920 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8h6k9" event={"ID":"1c121fea-270e-4763-aff7-53348028373d","Type":"ContainerStarted","Data":"403e7bc75a1f34113cf7be2bc1c39b69696e0eee96ca5bff24b6ba434c4c1c44"} Jul 30 00:07:03.985745 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:03.985524 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8h6k9" podStartSLOduration=4.012734997 podStartE2EDuration="4.985506532s" totalImagesPullingTime="900.548994ms" totalInitContainerRuntime="72.222541ms" isStatefulPod=true podCreationTimestamp="2026-07-30 00:06:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:07:03.983519782 +0000 UTC m=+311.486778041" watchObservedRunningTime="2026-07-30 00:07:03.985506532 +0000 UTC m=+311.488764792" Jul 30 00:07:03.986784 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:03.986762 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:07:04.195325 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:04.195285 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:05.485182 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:05.485091 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:05.578796 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:05.578764 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:05.689048 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:05.689016 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:05.973804 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:05.973766 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nx2v9" event={"ID":"146c3400-fc20-48bb-a456-e7b4f53f406f","Type":"ContainerStarted","Data":"c4b3bc6707aa43cbe537f70ea19238477dbb2f247bbaf5d1b45a6991e8b33f62"} Jul 30 00:07:05.991729 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:05.991656 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nx2v9" podStartSLOduration=33.966531599 podStartE2EDuration="47.991639685s" totalImagesPullingTime="14.025108086s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 00:06:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:07:05.990742001 +0000 UTC m=+313.494000262" watchObservedRunningTime="2026-07-30 00:07:05.991639685 +0000 UTC m=+313.494897944" Jul 30 00:07:06.387921 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:06.387891 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:06.488009 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:06.487977 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:06.799994 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:06.799910 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:07.481981 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:07.481953 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:07.582913 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:07.582883 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:12.793742 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:12.793707 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:12.893545 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:12.893519 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:18.368406 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:18.368371 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:18.421770 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:18.421747 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:19.109792 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:19.109762 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:19.164383 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:19.164356 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:21.385712 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:21.385683 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:21.436383 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:21.436358 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:22.231463 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:22.231430 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:22.330338 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:22.330309 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:36.390638 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:36.390606 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:36.441363 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:36.441334 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:07:51.885177 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:07:51.885056 2572 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" podUID="d6fc6f82-c5da-479b-a03b-6022fc8d678f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 30 00:08:01.884939 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:01.884900 2572 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" podUID="d6fc6f82-c5da-479b-a03b-6022fc8d678f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 30 00:08:06.798985 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:06.798945 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs\") pod \"network-metrics-daemon-nk989\" (UID: \"a38e0c1c-7133-40ff-9ba4-9a066da5c41e\") " pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:08:06.801333 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:06.801308 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a38e0c1c-7133-40ff-9ba4-9a066da5c41e-metrics-certs\") pod \"network-metrics-daemon-nk989\" (UID: \"a38e0c1c-7133-40ff-9ba4-9a066da5c41e\") " pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:08:06.989876 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:06.989847 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qpk4b\"" Jul 30 00:08:06.998024 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:06.998003 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nk989" Jul 30 00:08:07.118790 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:07.118755 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nk989"] Jul 30 00:08:07.121599 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:08:07.121576 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda38e0c1c_7133_40ff_9ba4_9a066da5c41e.slice/crio-fd2bc36f967d0e96b7392d165c4fadaa6dedd157974de2dcecea6d9fd108946a WatchSource:0}: Error finding container fd2bc36f967d0e96b7392d165c4fadaa6dedd157974de2dcecea6d9fd108946a: Status 404 returned error can't find the container with id fd2bc36f967d0e96b7392d165c4fadaa6dedd157974de2dcecea6d9fd108946a Jul 30 00:08:07.155575 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:07.155546 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nk989" event={"ID":"a38e0c1c-7133-40ff-9ba4-9a066da5c41e","Type":"ContainerStarted","Data":"fd2bc36f967d0e96b7392d165c4fadaa6dedd157974de2dcecea6d9fd108946a"} Jul 30 00:08:08.161037 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:08.161003 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nk989" event={"ID":"a38e0c1c-7133-40ff-9ba4-9a066da5c41e","Type":"ContainerStarted","Data":"22345e9e96186f41a9b88f67eab48528058cc8b1443ee2b2fe2204299fe4db97"} Jul 30 00:08:08.161385 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:08.161066 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nk989" event={"ID":"a38e0c1c-7133-40ff-9ba4-9a066da5c41e","Type":"ContainerStarted","Data":"0784b6479701f3c819428a111fda50e3e2b5d27d2d026f3ae954c36a03203b2f"} Jul 30 00:08:08.175910 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:08.175867 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nk989" podStartSLOduration=374.293713364 podStartE2EDuration="6m15.175849703s" totalImagesPullingTime="882.136339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:08:08.17499622 +0000 UTC m=+375.678254481" watchObservedRunningTime="2026-07-30 00:08:08.175849703 +0000 UTC m=+375.679107964" Jul 30 00:08:08.255006 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:08.254933 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:08:08.356561 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:08.356534 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:08:09.800135 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:09.800090 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:08:09.855494 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:09.855468 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:08:11.885406 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:11.885366 2572 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" podUID="d6fc6f82-c5da-479b-a03b-6022fc8d678f" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 30 00:08:11.885809 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:11.885432 2572 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" Jul 30 00:08:11.885904 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:11.885876 2572 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"a178f843ac52271629e1049486ffa9c80ac43692cfa4eaa15d39f6ade2ddedf2"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 30 00:08:11.885941 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:11.885924 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" podUID="d6fc6f82-c5da-479b-a03b-6022fc8d678f" containerName="service-proxy" containerID="cri-o://a178f843ac52271629e1049486ffa9c80ac43692cfa4eaa15d39f6ade2ddedf2" gracePeriod=30 Jul 30 00:08:12.176573 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:12.176484 2572 generic.go:356] "Generic (PLEG): container finished" podID="d6fc6f82-c5da-479b-a03b-6022fc8d678f" containerID="a178f843ac52271629e1049486ffa9c80ac43692cfa4eaa15d39f6ade2ddedf2" exitCode=2 Jul 30 00:08:12.176573 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:12.176525 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" event={"ID":"d6fc6f82-c5da-479b-a03b-6022fc8d678f","Type":"ContainerDied","Data":"a178f843ac52271629e1049486ffa9c80ac43692cfa4eaa15d39f6ade2ddedf2"} Jul 30 00:08:12.176573 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:12.176563 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5cc8795f79-hswhp" event={"ID":"d6fc6f82-c5da-479b-a03b-6022fc8d678f","Type":"ContainerStarted","Data":"ff1de195d8b0a7b9a574d357b170294609e91b0cb2a7dff947283c893898b332"} Jul 30 00:08:18.605399 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:18.605357 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:08:19.130896 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:19.130844 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:08:20.557327 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:20.557257 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:08:20.614798 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:20.614755 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:08:27.302611 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:27.302573 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:08:27.404389 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:27.404359 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:08:27.504418 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:27.504386 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:08:28.057259 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:28.057218 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:08:28.114527 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:28.114490 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:08:29.799712 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:29.799674 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:08:33.859162 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:33.859130 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:08:33.960702 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:33.960671 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:08:38.257745 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:38.257713 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:08:38.344625 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:38.344590 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:08:38.854228 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:38.854194 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:08:38.910039 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:38.909995 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:08:45.757382 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:45.757321 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:08:45.813085 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:45.813032 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:08:48.961633 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:48.961596 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:08:49.017336 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:49.017306 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:08:49.079097 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:49.079043 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:08:51.417506 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:51.417442 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:08:51.517585 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:08:51.517535 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:09:09.680490 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:09:09.680447 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:09:09.734323 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:09:09.734279 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:09:15.658221 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:09:15.658125 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:09:15.748957 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:09:15.748911 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:09:42.310179 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:09:42.310150 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:09:43.155096 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:09:43.155059 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:09:45.932926 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:09:45.932864 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:09:46.026056 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:09:46.026026 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:09:50.714756 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:09:50.714724 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:09:50.766512 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:09:50.766487 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:09:51.951849 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:09:51.951814 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:09:52.046230 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:09:52.046193 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:09:52.265661 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:09:52.265592 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:09:52.363078 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:09:52.363049 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:09:54.170675 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:09:54.170645 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:09:54.360783 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:09:54.360751 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:02.556075 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:02.556041 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:05.030197 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:05.030153 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:05.111262 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:05.111226 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:12.835127 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:12.835081 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:12.935387 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:12.935355 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:14.624453 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:14.624424 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:18.142610 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:18.142566 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:18.195196 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:18.195155 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:28.155687 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:28.155639 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:28.251651 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:28.251569 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:28.979493 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:28.979459 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:29.232782 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.232703 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:29.345034 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.345001 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-655d97b687-l8fnr"] Jul 30 00:10:29.347963 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.347947 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-655d97b687-l8fnr" Jul 30 00:10:29.349553 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.349524 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:29.352410 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.352392 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 30 00:10:29.352709 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.352695 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 30 00:10:29.352767 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.352736 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 30 00:10:29.352883 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.352868 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-59lxz\"" Jul 30 00:10:29.352940 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.352903 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 30 00:10:29.371337 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.371313 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-655d97b687-l8fnr"] Jul 30 00:10:29.484724 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.484649 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b446af54-7c30-437b-9faa-7e17c067316e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-655d97b687-l8fnr\" (UID: \"b446af54-7c30-437b-9faa-7e17c067316e\") " pod="opendatahub/opendatahub-operator-controller-manager-655d97b687-l8fnr" Jul 30 00:10:29.484724 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.484695 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl9nd\" (UniqueName: \"kubernetes.io/projected/b446af54-7c30-437b-9faa-7e17c067316e-kube-api-access-vl9nd\") pod \"opendatahub-operator-controller-manager-655d97b687-l8fnr\" (UID: \"b446af54-7c30-437b-9faa-7e17c067316e\") " pod="opendatahub/opendatahub-operator-controller-manager-655d97b687-l8fnr" Jul 30 00:10:29.484724 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.484719 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b446af54-7c30-437b-9faa-7e17c067316e-manifests\") pod \"opendatahub-operator-controller-manager-655d97b687-l8fnr\" (UID: \"b446af54-7c30-437b-9faa-7e17c067316e\") " pod="opendatahub/opendatahub-operator-controller-manager-655d97b687-l8fnr" Jul 30 00:10:29.484898 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.484764 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b446af54-7c30-437b-9faa-7e17c067316e-tmp\") pod \"opendatahub-operator-controller-manager-655d97b687-l8fnr\" (UID: \"b446af54-7c30-437b-9faa-7e17c067316e\") " pod="opendatahub/opendatahub-operator-controller-manager-655d97b687-l8fnr" Jul 30 00:10:29.484898 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.484778 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b446af54-7c30-437b-9faa-7e17c067316e-webhook-cert\") pod \"opendatahub-operator-controller-manager-655d97b687-l8fnr\" (UID: \"b446af54-7c30-437b-9faa-7e17c067316e\") " pod="opendatahub/opendatahub-operator-controller-manager-655d97b687-l8fnr" Jul 30 00:10:29.503954 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.503924 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:29.585054 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.585025 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b446af54-7c30-437b-9faa-7e17c067316e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-655d97b687-l8fnr\" (UID: \"b446af54-7c30-437b-9faa-7e17c067316e\") " pod="opendatahub/opendatahub-operator-controller-manager-655d97b687-l8fnr" Jul 30 00:10:29.585183 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.585064 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl9nd\" (UniqueName: \"kubernetes.io/projected/b446af54-7c30-437b-9faa-7e17c067316e-kube-api-access-vl9nd\") pod \"opendatahub-operator-controller-manager-655d97b687-l8fnr\" (UID: \"b446af54-7c30-437b-9faa-7e17c067316e\") " pod="opendatahub/opendatahub-operator-controller-manager-655d97b687-l8fnr" Jul 30 00:10:29.585183 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.585093 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b446af54-7c30-437b-9faa-7e17c067316e-manifests\") pod \"opendatahub-operator-controller-manager-655d97b687-l8fnr\" (UID: \"b446af54-7c30-437b-9faa-7e17c067316e\") " pod="opendatahub/opendatahub-operator-controller-manager-655d97b687-l8fnr" Jul 30 00:10:29.585183 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.585142 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b446af54-7c30-437b-9faa-7e17c067316e-tmp\") pod \"opendatahub-operator-controller-manager-655d97b687-l8fnr\" (UID: \"b446af54-7c30-437b-9faa-7e17c067316e\") " pod="opendatahub/opendatahub-operator-controller-manager-655d97b687-l8fnr" Jul 30 00:10:29.585183 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.585170 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b446af54-7c30-437b-9faa-7e17c067316e-webhook-cert\") pod \"opendatahub-operator-controller-manager-655d97b687-l8fnr\" (UID: \"b446af54-7c30-437b-9faa-7e17c067316e\") " pod="opendatahub/opendatahub-operator-controller-manager-655d97b687-l8fnr" Jul 30 00:10:29.585602 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.585577 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b446af54-7c30-437b-9faa-7e17c067316e-manifests\") pod \"opendatahub-operator-controller-manager-655d97b687-l8fnr\" (UID: \"b446af54-7c30-437b-9faa-7e17c067316e\") " pod="opendatahub/opendatahub-operator-controller-manager-655d97b687-l8fnr" Jul 30 00:10:29.587459 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.587436 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b446af54-7c30-437b-9faa-7e17c067316e-tmp\") pod \"opendatahub-operator-controller-manager-655d97b687-l8fnr\" (UID: \"b446af54-7c30-437b-9faa-7e17c067316e\") " pod="opendatahub/opendatahub-operator-controller-manager-655d97b687-l8fnr" Jul 30 00:10:29.587534 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.587511 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b446af54-7c30-437b-9faa-7e17c067316e-webhook-cert\") pod \"opendatahub-operator-controller-manager-655d97b687-l8fnr\" (UID: \"b446af54-7c30-437b-9faa-7e17c067316e\") " pod="opendatahub/opendatahub-operator-controller-manager-655d97b687-l8fnr" Jul 30 00:10:29.587670 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.587653 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b446af54-7c30-437b-9faa-7e17c067316e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-655d97b687-l8fnr\" (UID: \"b446af54-7c30-437b-9faa-7e17c067316e\") " pod="opendatahub/opendatahub-operator-controller-manager-655d97b687-l8fnr" Jul 30 00:10:29.593131 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.593092 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl9nd\" (UniqueName: \"kubernetes.io/projected/b446af54-7c30-437b-9faa-7e17c067316e-kube-api-access-vl9nd\") pod \"opendatahub-operator-controller-manager-655d97b687-l8fnr\" (UID: \"b446af54-7c30-437b-9faa-7e17c067316e\") " pod="opendatahub/opendatahub-operator-controller-manager-655d97b687-l8fnr" Jul 30 00:10:29.604040 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.604020 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:29.657673 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.657650 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-655d97b687-l8fnr" Jul 30 00:10:29.705574 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.705519 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:29.782335 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.782265 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-655d97b687-l8fnr"] Jul 30 00:10:29.786367 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:10:29.786341 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb446af54_7c30_437b_9faa_7e17c067316e.slice/crio-cac2e5c587dda3b365bc695b30ba1c573ffba964ce003d0101ea7eb6483a4baa WatchSource:0}: Error finding container cac2e5c587dda3b365bc695b30ba1c573ffba964ce003d0101ea7eb6483a4baa: Status 404 returned error can't find the container with id cac2e5c587dda3b365bc695b30ba1c573ffba964ce003d0101ea7eb6483a4baa Jul 30 00:10:29.806422 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.806402 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:29.906059 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:29.906037 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:30.005345 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:30.005320 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:30.632637 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:30.632557 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-655d97b687-l8fnr" event={"ID":"b446af54-7c30-437b-9faa-7e17c067316e","Type":"ContainerStarted","Data":"cac2e5c587dda3b365bc695b30ba1c573ffba964ce003d0101ea7eb6483a4baa"} Jul 30 00:10:33.646223 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:33.646190 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-655d97b687-l8fnr" event={"ID":"b446af54-7c30-437b-9faa-7e17c067316e","Type":"ContainerStarted","Data":"82515307f6c8fcf9e5adcf553ecb80ecb13513b36a940372876d787d75d4c544"} Jul 30 00:10:34.650650 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:34.650617 2572 generic.go:356] "Generic (PLEG): container finished" podID="b446af54-7c30-437b-9faa-7e17c067316e" containerID="82515307f6c8fcf9e5adcf553ecb80ecb13513b36a940372876d787d75d4c544" exitCode=0 Jul 30 00:10:34.651048 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:34.650674 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-655d97b687-l8fnr" event={"ID":"b446af54-7c30-437b-9faa-7e17c067316e","Type":"ContainerDied","Data":"82515307f6c8fcf9e5adcf553ecb80ecb13513b36a940372876d787d75d4c544"} Jul 30 00:10:35.656470 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:35.656435 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-655d97b687-l8fnr" event={"ID":"b446af54-7c30-437b-9faa-7e17c067316e","Type":"ContainerStarted","Data":"ad8c8d277de24edb6e1b548cdaae83c59972ea716323e15eec6ebc4a80e3961f"} Jul 30 00:10:35.656861 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:35.656563 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-655d97b687-l8fnr" Jul 30 00:10:35.692266 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:35.692209 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-655d97b687-l8fnr" podStartSLOduration=2.257409596 podStartE2EDuration="6.692189695s" totalImagesPullingTime="3.453493705s" totalInitContainerRuntime="981.286394ms" isStatefulPod=false podCreationTimestamp="2026-07-30 00:10:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:10:35.691621536 +0000 UTC m=+523.194879794" watchObservedRunningTime="2026-07-30 00:10:35.692189695 +0000 UTC m=+523.195447954" Jul 30 00:10:40.992214 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:40.992178 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:41.089583 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:41.089551 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:46.662523 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:46.662447 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-655d97b687-l8fnr" Jul 30 00:10:47.132870 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:47.132836 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5977989bd7-9xvmv"] Jul 30 00:10:47.135704 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:47.135685 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5977989bd7-9xvmv" Jul 30 00:10:47.140120 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:47.140083 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 30 00:10:47.140290 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:47.140123 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 30 00:10:47.140290 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:47.140140 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 30 00:10:47.140290 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:47.140161 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-l58xp\"" Jul 30 00:10:47.140290 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:47.140102 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 30 00:10:47.140290 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:47.140256 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 30 00:10:47.147899 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:47.147871 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5977989bd7-9xvmv"] Jul 30 00:10:47.209238 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:47.209207 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e88b0711-0653-42eb-a6e0-1272d5e62b8f-metrics-cert\") pod \"lws-controller-manager-5977989bd7-9xvmv\" (UID: \"e88b0711-0653-42eb-a6e0-1272d5e62b8f\") " pod="openshift-lws-operator/lws-controller-manager-5977989bd7-9xvmv" Jul 30 00:10:47.209413 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:47.209269 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmfhv\" (UniqueName: \"kubernetes.io/projected/e88b0711-0653-42eb-a6e0-1272d5e62b8f-kube-api-access-cmfhv\") pod \"lws-controller-manager-5977989bd7-9xvmv\" (UID: \"e88b0711-0653-42eb-a6e0-1272d5e62b8f\") " pod="openshift-lws-operator/lws-controller-manager-5977989bd7-9xvmv" Jul 30 00:10:47.209413 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:47.209321 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e88b0711-0653-42eb-a6e0-1272d5e62b8f-manager-config\") pod \"lws-controller-manager-5977989bd7-9xvmv\" (UID: \"e88b0711-0653-42eb-a6e0-1272d5e62b8f\") " pod="openshift-lws-operator/lws-controller-manager-5977989bd7-9xvmv" Jul 30 00:10:47.209413 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:47.209384 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e88b0711-0653-42eb-a6e0-1272d5e62b8f-cert\") pod \"lws-controller-manager-5977989bd7-9xvmv\" (UID: \"e88b0711-0653-42eb-a6e0-1272d5e62b8f\") " pod="openshift-lws-operator/lws-controller-manager-5977989bd7-9xvmv" Jul 30 00:10:47.310732 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:47.310687 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e88b0711-0653-42eb-a6e0-1272d5e62b8f-metrics-cert\") pod \"lws-controller-manager-5977989bd7-9xvmv\" (UID: \"e88b0711-0653-42eb-a6e0-1272d5e62b8f\") " pod="openshift-lws-operator/lws-controller-manager-5977989bd7-9xvmv" Jul 30 00:10:47.310903 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:47.310822 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmfhv\" (UniqueName: \"kubernetes.io/projected/e88b0711-0653-42eb-a6e0-1272d5e62b8f-kube-api-access-cmfhv\") pod \"lws-controller-manager-5977989bd7-9xvmv\" (UID: \"e88b0711-0653-42eb-a6e0-1272d5e62b8f\") " pod="openshift-lws-operator/lws-controller-manager-5977989bd7-9xvmv" Jul 30 00:10:47.310903 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:47.310857 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e88b0711-0653-42eb-a6e0-1272d5e62b8f-manager-config\") pod \"lws-controller-manager-5977989bd7-9xvmv\" (UID: \"e88b0711-0653-42eb-a6e0-1272d5e62b8f\") " pod="openshift-lws-operator/lws-controller-manager-5977989bd7-9xvmv" Jul 30 00:10:47.310903 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:47.310884 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e88b0711-0653-42eb-a6e0-1272d5e62b8f-cert\") pod \"lws-controller-manager-5977989bd7-9xvmv\" (UID: \"e88b0711-0653-42eb-a6e0-1272d5e62b8f\") " pod="openshift-lws-operator/lws-controller-manager-5977989bd7-9xvmv" Jul 30 00:10:47.311540 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:47.311512 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e88b0711-0653-42eb-a6e0-1272d5e62b8f-manager-config\") pod \"lws-controller-manager-5977989bd7-9xvmv\" (UID: \"e88b0711-0653-42eb-a6e0-1272d5e62b8f\") " pod="openshift-lws-operator/lws-controller-manager-5977989bd7-9xvmv" Jul 30 00:10:47.313178 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:47.313151 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e88b0711-0653-42eb-a6e0-1272d5e62b8f-cert\") pod \"lws-controller-manager-5977989bd7-9xvmv\" (UID: \"e88b0711-0653-42eb-a6e0-1272d5e62b8f\") " pod="openshift-lws-operator/lws-controller-manager-5977989bd7-9xvmv" Jul 30 00:10:47.313356 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:47.313334 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e88b0711-0653-42eb-a6e0-1272d5e62b8f-metrics-cert\") pod \"lws-controller-manager-5977989bd7-9xvmv\" (UID: \"e88b0711-0653-42eb-a6e0-1272d5e62b8f\") " pod="openshift-lws-operator/lws-controller-manager-5977989bd7-9xvmv" Jul 30 00:10:47.319983 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:47.319958 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmfhv\" (UniqueName: \"kubernetes.io/projected/e88b0711-0653-42eb-a6e0-1272d5e62b8f-kube-api-access-cmfhv\") pod \"lws-controller-manager-5977989bd7-9xvmv\" (UID: \"e88b0711-0653-42eb-a6e0-1272d5e62b8f\") " pod="openshift-lws-operator/lws-controller-manager-5977989bd7-9xvmv" Jul 30 00:10:47.445152 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:47.445040 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5977989bd7-9xvmv" Jul 30 00:10:47.573141 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:47.573100 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5977989bd7-9xvmv"] Jul 30 00:10:47.575630 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:10:47.575585 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode88b0711_0653_42eb_a6e0_1272d5e62b8f.slice/crio-905f211dcb3ff1922ed9b68f4cdff6e3d93f797993f7851694229c0ed2e3b98d WatchSource:0}: Error finding container 905f211dcb3ff1922ed9b68f4cdff6e3d93f797993f7851694229c0ed2e3b98d: Status 404 returned error can't find the container with id 905f211dcb3ff1922ed9b68f4cdff6e3d93f797993f7851694229c0ed2e3b98d Jul 30 00:10:47.696494 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:47.696407 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5977989bd7-9xvmv" event={"ID":"e88b0711-0653-42eb-a6e0-1272d5e62b8f","Type":"ContainerStarted","Data":"905f211dcb3ff1922ed9b68f4cdff6e3d93f797993f7851694229c0ed2e3b98d"} Jul 30 00:10:48.598885 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:48.598830 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:48.740896 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:48.740803 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:49.248911 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:49.248863 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:49.500882 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:49.500789 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:49.596906 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:49.596871 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:49.693087 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:49.693047 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:49.791526 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:49.791448 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:49.895287 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:49.895254 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:50.450960 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:50.450928 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:50.595571 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:50.595539 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:50.685058 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:50.685027 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:50.710202 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:50.710167 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5977989bd7-9xvmv" event={"ID":"e88b0711-0653-42eb-a6e0-1272d5e62b8f","Type":"ContainerStarted","Data":"b3b64d605e2a4c43e49c5a0c1bed0239b1211b3a8afa46b7151dd68d3b6e018b"} Jul 30 00:10:50.710369 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:50.710263 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5977989bd7-9xvmv" Jul 30 00:10:50.729491 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:50.729445 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5977989bd7-9xvmv" podStartSLOduration=1.068210937 podStartE2EDuration="3.729430583s" totalImagesPullingTime="2.661219646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:10:50.728224901 +0000 UTC m=+538.231483178" watchObservedRunningTime="2026-07-30 00:10:50.729430583 +0000 UTC m=+538.232688841" Jul 30 00:10:50.759650 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:50.759618 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:51.662127 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:51.662050 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:56.634771 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:56.634735 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:56.742467 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:56.742435 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:56.953782 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:56.953704 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8"] Jul 30 00:10:56.958480 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:56.958460 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:56.962154 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:56.962125 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 30 00:10:56.962408 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:56.962387 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-f2tt7\"" Jul 30 00:10:56.962520 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:56.962408 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 30 00:10:56.962520 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:56.962457 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 30 00:10:56.976656 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:56.976621 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8"] Jul 30 00:10:57.000734 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.000701 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:57.087181 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.087142 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937f26e2-fce2-4ab8-8c74-a71670a333cd-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8\" (UID: \"937f26e2-fce2-4ab8-8c74-a71670a333cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:57.087181 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.087188 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937f26e2-fce2-4ab8-8c74-a71670a333cd-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8\" (UID: \"937f26e2-fce2-4ab8-8c74-a71670a333cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:57.087418 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.087231 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937f26e2-fce2-4ab8-8c74-a71670a333cd-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8\" (UID: \"937f26e2-fce2-4ab8-8c74-a71670a333cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:57.087418 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.087267 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937f26e2-fce2-4ab8-8c74-a71670a333cd-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8\" (UID: \"937f26e2-fce2-4ab8-8c74-a71670a333cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:57.087418 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.087296 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937f26e2-fce2-4ab8-8c74-a71670a333cd-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8\" (UID: \"937f26e2-fce2-4ab8-8c74-a71670a333cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:57.087418 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.087357 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937f26e2-fce2-4ab8-8c74-a71670a333cd-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8\" (UID: \"937f26e2-fce2-4ab8-8c74-a71670a333cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:57.087418 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.087382 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kllqx\" (UniqueName: \"kubernetes.io/projected/937f26e2-fce2-4ab8-8c74-a71670a333cd-kube-api-access-kllqx\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8\" (UID: \"937f26e2-fce2-4ab8-8c74-a71670a333cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:57.087418 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.087411 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937f26e2-fce2-4ab8-8c74-a71670a333cd-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8\" (UID: \"937f26e2-fce2-4ab8-8c74-a71670a333cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:57.087612 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.087429 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937f26e2-fce2-4ab8-8c74-a71670a333cd-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8\" (UID: \"937f26e2-fce2-4ab8-8c74-a71670a333cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:57.142393 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.142364 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl"] Jul 30 00:10:57.145335 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.145319 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl" Jul 30 00:10:57.148049 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.148028 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 30 00:10:57.148324 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.148301 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ggssx\"" Jul 30 00:10:57.148432 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.148333 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 30 00:10:57.148530 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.148518 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 30 00:10:57.158371 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.158348 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:57.162653 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.162621 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl"] Jul 30 00:10:57.188429 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.188395 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937f26e2-fce2-4ab8-8c74-a71670a333cd-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8\" (UID: \"937f26e2-fce2-4ab8-8c74-a71670a333cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:57.188628 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.188440 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937f26e2-fce2-4ab8-8c74-a71670a333cd-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8\" (UID: \"937f26e2-fce2-4ab8-8c74-a71670a333cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:57.188628 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.188477 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937f26e2-fce2-4ab8-8c74-a71670a333cd-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8\" (UID: \"937f26e2-fce2-4ab8-8c74-a71670a333cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:57.188628 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.188537 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937f26e2-fce2-4ab8-8c74-a71670a333cd-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8\" (UID: \"937f26e2-fce2-4ab8-8c74-a71670a333cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:57.188628 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.188563 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kllqx\" (UniqueName: \"kubernetes.io/projected/937f26e2-fce2-4ab8-8c74-a71670a333cd-kube-api-access-kllqx\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8\" (UID: \"937f26e2-fce2-4ab8-8c74-a71670a333cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:57.188628 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.188592 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937f26e2-fce2-4ab8-8c74-a71670a333cd-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8\" (UID: \"937f26e2-fce2-4ab8-8c74-a71670a333cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:57.189022 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.188892 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937f26e2-fce2-4ab8-8c74-a71670a333cd-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8\" (UID: \"937f26e2-fce2-4ab8-8c74-a71670a333cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:57.189022 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.188958 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937f26e2-fce2-4ab8-8c74-a71670a333cd-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8\" (UID: \"937f26e2-fce2-4ab8-8c74-a71670a333cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:57.189022 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.189004 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937f26e2-fce2-4ab8-8c74-a71670a333cd-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8\" (UID: \"937f26e2-fce2-4ab8-8c74-a71670a333cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:57.189022 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.189008 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937f26e2-fce2-4ab8-8c74-a71670a333cd-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8\" (UID: \"937f26e2-fce2-4ab8-8c74-a71670a333cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:57.189242 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.189036 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937f26e2-fce2-4ab8-8c74-a71670a333cd-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8\" (UID: \"937f26e2-fce2-4ab8-8c74-a71670a333cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:57.189242 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.189048 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937f26e2-fce2-4ab8-8c74-a71670a333cd-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8\" (UID: \"937f26e2-fce2-4ab8-8c74-a71670a333cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:57.189373 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.189355 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937f26e2-fce2-4ab8-8c74-a71670a333cd-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8\" (UID: \"937f26e2-fce2-4ab8-8c74-a71670a333cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:57.190033 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.190006 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937f26e2-fce2-4ab8-8c74-a71670a333cd-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8\" (UID: \"937f26e2-fce2-4ab8-8c74-a71670a333cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:57.191332 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.191308 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937f26e2-fce2-4ab8-8c74-a71670a333cd-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8\" (UID: \"937f26e2-fce2-4ab8-8c74-a71670a333cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:57.191814 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.191798 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937f26e2-fce2-4ab8-8c74-a71670a333cd-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8\" (UID: \"937f26e2-fce2-4ab8-8c74-a71670a333cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:57.239428 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.239359 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kllqx\" (UniqueName: \"kubernetes.io/projected/937f26e2-fce2-4ab8-8c74-a71670a333cd-kube-api-access-kllqx\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8\" (UID: \"937f26e2-fce2-4ab8-8c74-a71670a333cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:57.241307 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.241288 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937f26e2-fce2-4ab8-8c74-a71670a333cd-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8\" (UID: \"937f26e2-fce2-4ab8-8c74-a71670a333cd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:57.269323 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.269280 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:10:57.289784 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.289727 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5826506a-70dc-4388-923e-cb7f5df65924-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-w62gl\" (UID: \"5826506a-70dc-4388-923e-cb7f5df65924\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl" Jul 30 00:10:57.289784 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.289780 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5826506a-70dc-4388-923e-cb7f5df65924-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-w62gl\" (UID: \"5826506a-70dc-4388-923e-cb7f5df65924\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl" Jul 30 00:10:57.290009 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.289816 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5826506a-70dc-4388-923e-cb7f5df65924-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-w62gl\" (UID: \"5826506a-70dc-4388-923e-cb7f5df65924\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl" Jul 30 00:10:57.290009 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.289847 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn5cb\" (UniqueName: \"kubernetes.io/projected/5826506a-70dc-4388-923e-cb7f5df65924-kube-api-access-sn5cb\") pod \"istiod-openshift-gateway-5677ccb49b-w62gl\" (UID: \"5826506a-70dc-4388-923e-cb7f5df65924\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl" Jul 30 00:10:57.290009 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.289915 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5826506a-70dc-4388-923e-cb7f5df65924-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-w62gl\" (UID: \"5826506a-70dc-4388-923e-cb7f5df65924\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl" Jul 30 00:10:57.290009 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.289963 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5826506a-70dc-4388-923e-cb7f5df65924-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-w62gl\" (UID: \"5826506a-70dc-4388-923e-cb7f5df65924\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl" Jul 30 00:10:57.290009 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.290005 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5826506a-70dc-4388-923e-cb7f5df65924-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-w62gl\" (UID: \"5826506a-70dc-4388-923e-cb7f5df65924\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl" Jul 30 00:10:57.373643 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.373608 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:57.390653 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.390618 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5826506a-70dc-4388-923e-cb7f5df65924-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-w62gl\" (UID: \"5826506a-70dc-4388-923e-cb7f5df65924\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl" Jul 30 00:10:57.390653 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.390657 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5826506a-70dc-4388-923e-cb7f5df65924-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-w62gl\" (UID: \"5826506a-70dc-4388-923e-cb7f5df65924\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl" Jul 30 00:10:57.390904 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.390680 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5826506a-70dc-4388-923e-cb7f5df65924-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-w62gl\" (UID: \"5826506a-70dc-4388-923e-cb7f5df65924\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl" Jul 30 00:10:57.390904 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.390742 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5826506a-70dc-4388-923e-cb7f5df65924-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-w62gl\" (UID: \"5826506a-70dc-4388-923e-cb7f5df65924\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl" Jul 30 00:10:57.390904 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.390766 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5826506a-70dc-4388-923e-cb7f5df65924-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-w62gl\" (UID: \"5826506a-70dc-4388-923e-cb7f5df65924\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl" Jul 30 00:10:57.390904 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.390794 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5826506a-70dc-4388-923e-cb7f5df65924-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-w62gl\" (UID: \"5826506a-70dc-4388-923e-cb7f5df65924\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl" Jul 30 00:10:57.390904 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.390823 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn5cb\" (UniqueName: \"kubernetes.io/projected/5826506a-70dc-4388-923e-cb7f5df65924-kube-api-access-sn5cb\") pod \"istiod-openshift-gateway-5677ccb49b-w62gl\" (UID: \"5826506a-70dc-4388-923e-cb7f5df65924\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl" Jul 30 00:10:57.391630 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.391601 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5826506a-70dc-4388-923e-cb7f5df65924-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-w62gl\" (UID: \"5826506a-70dc-4388-923e-cb7f5df65924\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl" Jul 30 00:10:57.393076 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.393045 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5826506a-70dc-4388-923e-cb7f5df65924-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-w62gl\" (UID: \"5826506a-70dc-4388-923e-cb7f5df65924\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl" Jul 30 00:10:57.393301 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.393277 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5826506a-70dc-4388-923e-cb7f5df65924-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-w62gl\" (UID: \"5826506a-70dc-4388-923e-cb7f5df65924\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl" Jul 30 00:10:57.393391 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.393279 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5826506a-70dc-4388-923e-cb7f5df65924-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-w62gl\" (UID: \"5826506a-70dc-4388-923e-cb7f5df65924\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl" Jul 30 00:10:57.393391 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.393379 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5826506a-70dc-4388-923e-cb7f5df65924-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-w62gl\" (UID: \"5826506a-70dc-4388-923e-cb7f5df65924\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl" Jul 30 00:10:57.404070 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.404042 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn5cb\" (UniqueName: \"kubernetes.io/projected/5826506a-70dc-4388-923e-cb7f5df65924-kube-api-access-sn5cb\") pod \"istiod-openshift-gateway-5677ccb49b-w62gl\" (UID: \"5826506a-70dc-4388-923e-cb7f5df65924\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl" Jul 30 00:10:57.404221 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.404183 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5826506a-70dc-4388-923e-cb7f5df65924-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-w62gl\" (UID: \"5826506a-70dc-4388-923e-cb7f5df65924\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl" Jul 30 00:10:57.411039 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.411017 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8"] Jul 30 00:10:57.413199 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:10:57.413169 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937f26e2_fce2_4ab8_8c74_a71670a333cd.slice/crio-bab3bd8527761f11c78a77ecb8ad9bbf42df41d14fda1218f2ca3f1e682dbf56 WatchSource:0}: Error finding container bab3bd8527761f11c78a77ecb8ad9bbf42df41d14fda1218f2ca3f1e682dbf56: Status 404 returned error can't find the container with id bab3bd8527761f11c78a77ecb8ad9bbf42df41d14fda1218f2ca3f1e682dbf56 Jul 30 00:10:57.454329 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.454290 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl" Jul 30 00:10:57.599117 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.599037 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl"] Jul 30 00:10:57.601292 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:10:57.601264 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5826506a_70dc_4388_923e_cb7f5df65924.slice/crio-09b1bbe79b8d8b80f8422a9e40e783d6eda4850fe325a04f0114b6169fd7c7d7 WatchSource:0}: Error finding container 09b1bbe79b8d8b80f8422a9e40e783d6eda4850fe325a04f0114b6169fd7c7d7: Status 404 returned error can't find the container with id 09b1bbe79b8d8b80f8422a9e40e783d6eda4850fe325a04f0114b6169fd7c7d7 Jul 30 00:10:57.735059 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.735026 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl" event={"ID":"5826506a-70dc-4388-923e-cb7f5df65924","Type":"ContainerStarted","Data":"09b1bbe79b8d8b80f8422a9e40e783d6eda4850fe325a04f0114b6169fd7c7d7"} Jul 30 00:10:57.736148 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:57.736101 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" event={"ID":"937f26e2-fce2-4ab8-8c74-a71670a333cd","Type":"ContainerStarted","Data":"bab3bd8527761f11c78a77ecb8ad9bbf42df41d14fda1218f2ca3f1e682dbf56"} Jul 30 00:10:59.794184 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:59.794064 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:10:59.945175 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:10:59.945085 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:11:00.522029 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:11:00.521994 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:11:01.102302 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:11:01.102258 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:11:01.312899 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:11:01.312863 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:11:01.405467 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:11:01.405420 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:11:01.405554 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:11:01.405499 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:11:01.408388 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:11:01.408359 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:11:01.408469 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:11:01.408432 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:11:01.408531 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:11:01.408475 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:11:01.429324 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:11:01.429300 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:11:01.716233 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:11:01.716155 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5977989bd7-9xvmv" Jul 30 00:11:01.756122 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:11:01.756077 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl" event={"ID":"5826506a-70dc-4388-923e-cb7f5df65924","Type":"ContainerStarted","Data":"7f02d21b53ec61ee133d25d5e499858bf49da01e61472ff528bfcd8f253fe3a0"} Jul 30 00:11:01.756306 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:11:01.756281 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl" Jul 30 00:11:01.758447 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:11:01.758423 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" event={"ID":"937f26e2-fce2-4ab8-8c74-a71670a333cd","Type":"ContainerStarted","Data":"c4bc86d1b5b888993ab62a5785320d85ae97fcdb9e2a7effd6dbef7ae426f16a"} Jul 30 00:11:01.800895 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:11:01.800810 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl" podStartSLOduration=0.998883659 podStartE2EDuration="4.800788537s" totalImagesPullingTime="3.801904878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:10:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:11:01.800667846 +0000 UTC m=+549.303926106" watchObservedRunningTime="2026-07-30 00:11:01.800788537 +0000 UTC m=+549.304046798" Jul 30 00:11:01.841745 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:11:01.841678 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" podStartSLOduration=1.84883057 podStartE2EDuration="5.841380509s" totalImagesPullingTime="3.992549939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:10:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:11:01.840729613 +0000 UTC m=+549.343987873" watchObservedRunningTime="2026-07-30 00:11:01.841380509 +0000 UTC m=+549.344638770" Jul 30 00:11:02.269897 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:11:02.269862 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:11:02.274653 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:11:02.274624 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:11:02.762774 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:11:02.762742 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:11:02.763731 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:11:02.763710 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8" Jul 30 00:11:02.764438 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:11:02.764421 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-w62gl" Jul 30 00:11:50.042362 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:11:50.042322 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:11:51.748396 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:11:51.748360 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:11:56.857823 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:11:56.857790 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:11:56.936326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:11:56.936289 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:17.963537 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:17.963435 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:18.063372 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:18.063340 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:20.273906 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:20.273877 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:20.887289 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:20.887250 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:25.601952 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:25.601915 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:25.733330 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:25.733295 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:27.782323 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:27.782289 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:28.285239 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:28.285190 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:29.430689 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:29.430657 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:30.942714 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:30.942679 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:32.882220 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:32.882192 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:34.717749 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:34.717716 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:35.876569 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:35.876536 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:35.978381 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:35.978305 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:36.278800 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:36.278721 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:36.382838 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:36.382806 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:40.108641 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:40.108609 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:40.198248 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:40.198213 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:41.348166 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:41.348132 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:41.715475 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:41.715376 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:43.882643 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:43.882609 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:43.983888 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:43.983857 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:50.711441 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:50.711411 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:50.823354 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:50.823321 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:54.467484 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:54.467440 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:54.522777 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:54.522751 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:54.684380 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:54.684338 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:54.787520 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:54.787429 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:56.263249 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:56.263215 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:12:56.364117 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:12:56.364078 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:01.367255 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:01.367218 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:01.469515 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:01.469482 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:08.386433 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:08.386396 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:08.488644 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:08.488610 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:12.195294 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:12.195258 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:12.309759 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:12.309722 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:12.597193 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:12.597164 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:12.658160 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:12.658133 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:15.205288 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:15.205238 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:15.252939 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:15.252908 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:20.601769 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:20.601733 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:20.662810 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:20.662778 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:27.218711 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:27.218683 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:27.330713 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:27.330680 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:29.366272 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:29.366227 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:29.466062 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:29.466031 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:29.590190 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:29.590150 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:29.649194 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:29.649072 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:30.105197 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:30.105165 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:30.155298 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:30.155257 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:32.351319 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:32.351274 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:32.449159 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:32.449126 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:40.237717 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:40.237670 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:40.339065 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:40.339035 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:40.392008 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:40.391972 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:44.302482 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:44.302438 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:44.361728 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:44.361695 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:44.412201 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:44.412157 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:44.513059 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:44.513026 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:44.579463 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:44.579429 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:45.078572 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:45.078509 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:45.132573 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:45.132528 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:48.645610 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:48.645562 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:13:59.431852 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:13:59.431816 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:14:03.712258 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:03.712210 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:14:03.928009 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:03.927977 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:14:04.197579 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:04.197544 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4dmtb"] Jul 30 00:14:04.201996 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:04.201958 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4dmtb" Jul 30 00:14:04.204833 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:04.204803 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 30 00:14:04.204981 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:04.204860 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-mpwbs\"" Jul 30 00:14:04.204981 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:04.204871 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 30 00:14:04.205897 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:04.205882 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 30 00:14:04.210607 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:04.210575 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4dmtb"] Jul 30 00:14:04.229158 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:04.229128 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:14:04.291618 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:04.291580 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rb96\" (UniqueName: \"kubernetes.io/projected/7a9e81dd-5d22-4171-8734-6e0b8aafe115-kube-api-access-2rb96\") pod \"postgres-69fb4594cf-4dmtb\" (UID: \"7a9e81dd-5d22-4171-8734-6e0b8aafe115\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4dmtb" Jul 30 00:14:04.291618 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:04.291621 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7a9e81dd-5d22-4171-8734-6e0b8aafe115-data\") pod \"postgres-69fb4594cf-4dmtb\" (UID: \"7a9e81dd-5d22-4171-8734-6e0b8aafe115\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4dmtb" Jul 30 00:14:04.392426 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:04.392387 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rb96\" (UniqueName: \"kubernetes.io/projected/7a9e81dd-5d22-4171-8734-6e0b8aafe115-kube-api-access-2rb96\") pod \"postgres-69fb4594cf-4dmtb\" (UID: \"7a9e81dd-5d22-4171-8734-6e0b8aafe115\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4dmtb" Jul 30 00:14:04.392426 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:04.392433 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7a9e81dd-5d22-4171-8734-6e0b8aafe115-data\") pod \"postgres-69fb4594cf-4dmtb\" (UID: \"7a9e81dd-5d22-4171-8734-6e0b8aafe115\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4dmtb" Jul 30 00:14:04.392814 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:04.392794 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7a9e81dd-5d22-4171-8734-6e0b8aafe115-data\") pod \"postgres-69fb4594cf-4dmtb\" (UID: \"7a9e81dd-5d22-4171-8734-6e0b8aafe115\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4dmtb" Jul 30 00:14:04.400163 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:04.400141 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rb96\" (UniqueName: \"kubernetes.io/projected/7a9e81dd-5d22-4171-8734-6e0b8aafe115-kube-api-access-2rb96\") pod \"postgres-69fb4594cf-4dmtb\" (UID: \"7a9e81dd-5d22-4171-8734-6e0b8aafe115\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4dmtb" Jul 30 00:14:04.513816 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:04.513731 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4dmtb" Jul 30 00:14:04.639571 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:04.639538 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4dmtb"] Jul 30 00:14:04.641653 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:14:04.641624 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a9e81dd_5d22_4171_8734_6e0b8aafe115.slice/crio-36a8f204fa124f391b2549e8922b7691402eb7e914c187ad2259419c4aebcb15 WatchSource:0}: Error finding container 36a8f204fa124f391b2549e8922b7691402eb7e914c187ad2259419c4aebcb15: Status 404 returned error can't find the container with id 36a8f204fa124f391b2549e8922b7691402eb7e914c187ad2259419c4aebcb15 Jul 30 00:14:04.642841 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:04.642824 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 00:14:05.401652 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:05.401615 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4dmtb" event={"ID":"7a9e81dd-5d22-4171-8734-6e0b8aafe115","Type":"ContainerStarted","Data":"36a8f204fa124f391b2549e8922b7691402eb7e914c187ad2259419c4aebcb15"} Jul 30 00:14:10.423037 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:10.422946 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4dmtb" event={"ID":"7a9e81dd-5d22-4171-8734-6e0b8aafe115","Type":"ContainerStarted","Data":"44073c3a07d3db31804797288970994b0f5ac757b9aca597a97ae1381c4a3526"} Jul 30 00:14:10.423435 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:10.423058 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4dmtb" Jul 30 00:14:10.439431 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:10.439378 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4dmtb" podStartSLOduration=0.982532641 podStartE2EDuration="6.439361233s" totalImagesPullingTime="5.456828592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:14:10.438005228 +0000 UTC m=+737.941263488" watchObservedRunningTime="2026-07-30 00:14:10.439361233 +0000 UTC m=+737.942619493" Jul 30 00:14:16.455550 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:16.455517 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4dmtb" Jul 30 00:14:16.797021 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:16.796929 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:14:16.900572 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:16.900537 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:14:24.422462 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:24.422398 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:14:24.474177 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:24.474140 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:14:27.343854 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:27.343820 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:14:27.442100 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:27.442068 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:14:58.924362 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:58.924320 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:14:59.022807 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:14:59.022776 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:15:00.441496 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:00.441465 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:15:00.567731 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:00.567697 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8667dbffc5-nm7rm"] Jul 30 00:15:00.570671 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:00.570654 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8667dbffc5-nm7rm" Jul 30 00:15:00.573171 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:00.573153 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wft5v\"" Jul 30 00:15:00.573289 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:00.573187 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 30 00:15:00.580168 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:00.580144 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8667dbffc5-nm7rm"] Jul 30 00:15:00.648979 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:00.648947 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lssfz\" (UniqueName: \"kubernetes.io/projected/d70e206a-eb89-4d5b-83e9-e08ddbdd8d35-kube-api-access-lssfz\") pod \"maas-api-8667dbffc5-nm7rm\" (UID: \"d70e206a-eb89-4d5b-83e9-e08ddbdd8d35\") " pod="odh-ai-gateway-infra/maas-api-8667dbffc5-nm7rm" Jul 30 00:15:00.649172 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:00.649015 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d70e206a-eb89-4d5b-83e9-e08ddbdd8d35-maas-api-tls\") pod \"maas-api-8667dbffc5-nm7rm\" (UID: \"d70e206a-eb89-4d5b-83e9-e08ddbdd8d35\") " pod="odh-ai-gateway-infra/maas-api-8667dbffc5-nm7rm" Jul 30 00:15:00.649172 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:00.649056 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d70e206a-eb89-4d5b-83e9-e08ddbdd8d35-openshift-service-ca\") pod \"maas-api-8667dbffc5-nm7rm\" (UID: \"d70e206a-eb89-4d5b-83e9-e08ddbdd8d35\") " pod="odh-ai-gateway-infra/maas-api-8667dbffc5-nm7rm" Jul 30 00:15:00.749475 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:00.749385 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d70e206a-eb89-4d5b-83e9-e08ddbdd8d35-maas-api-tls\") pod \"maas-api-8667dbffc5-nm7rm\" (UID: \"d70e206a-eb89-4d5b-83e9-e08ddbdd8d35\") " pod="odh-ai-gateway-infra/maas-api-8667dbffc5-nm7rm" Jul 30 00:15:00.749475 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:00.749434 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d70e206a-eb89-4d5b-83e9-e08ddbdd8d35-openshift-service-ca\") pod \"maas-api-8667dbffc5-nm7rm\" (UID: \"d70e206a-eb89-4d5b-83e9-e08ddbdd8d35\") " pod="odh-ai-gateway-infra/maas-api-8667dbffc5-nm7rm" Jul 30 00:15:00.749475 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:00.749470 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lssfz\" (UniqueName: \"kubernetes.io/projected/d70e206a-eb89-4d5b-83e9-e08ddbdd8d35-kube-api-access-lssfz\") pod \"maas-api-8667dbffc5-nm7rm\" (UID: \"d70e206a-eb89-4d5b-83e9-e08ddbdd8d35\") " pod="odh-ai-gateway-infra/maas-api-8667dbffc5-nm7rm" Jul 30 00:15:00.749738 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:15:00.749548 2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 30 00:15:00.749738 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:15:00.749623 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d70e206a-eb89-4d5b-83e9-e08ddbdd8d35-maas-api-tls podName:d70e206a-eb89-4d5b-83e9-e08ddbdd8d35 nodeName:}" failed. No retries permitted until 2026-07-30 00:15:01.249603905 +0000 UTC m=+788.752862143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d70e206a-eb89-4d5b-83e9-e08ddbdd8d35-maas-api-tls") pod "maas-api-8667dbffc5-nm7rm" (UID: "d70e206a-eb89-4d5b-83e9-e08ddbdd8d35") : secret "maas-api-serving-cert" not found Jul 30 00:15:00.750128 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:00.750091 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d70e206a-eb89-4d5b-83e9-e08ddbdd8d35-openshift-service-ca\") pod \"maas-api-8667dbffc5-nm7rm\" (UID: \"d70e206a-eb89-4d5b-83e9-e08ddbdd8d35\") " pod="odh-ai-gateway-infra/maas-api-8667dbffc5-nm7rm" Jul 30 00:15:00.758195 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:00.758171 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lssfz\" (UniqueName: \"kubernetes.io/projected/d70e206a-eb89-4d5b-83e9-e08ddbdd8d35-kube-api-access-lssfz\") pod \"maas-api-8667dbffc5-nm7rm\" (UID: \"d70e206a-eb89-4d5b-83e9-e08ddbdd8d35\") " pod="odh-ai-gateway-infra/maas-api-8667dbffc5-nm7rm" Jul 30 00:15:00.924256 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:00.924224 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:15:01.040010 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:01.039924 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:15:01.190150 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:01.190120 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:15:01.253024 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:01.252994 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d70e206a-eb89-4d5b-83e9-e08ddbdd8d35-maas-api-tls\") pod \"maas-api-8667dbffc5-nm7rm\" (UID: \"d70e206a-eb89-4d5b-83e9-e08ddbdd8d35\") " pod="odh-ai-gateway-infra/maas-api-8667dbffc5-nm7rm" Jul 30 00:15:01.255382 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:01.255355 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d70e206a-eb89-4d5b-83e9-e08ddbdd8d35-maas-api-tls\") pod \"maas-api-8667dbffc5-nm7rm\" (UID: \"d70e206a-eb89-4d5b-83e9-e08ddbdd8d35\") " pod="odh-ai-gateway-infra/maas-api-8667dbffc5-nm7rm" Jul 30 00:15:01.482184 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:01.482143 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8667dbffc5-nm7rm" Jul 30 00:15:01.603444 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:01.603418 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8667dbffc5-nm7rm"] Jul 30 00:15:01.606024 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:15:01.605987 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd70e206a_eb89_4d5b_83e9_e08ddbdd8d35.slice/crio-0186ae1be4a1224bc093f72e3346486b74302d4b7d41c2f061e2a7e8bb37d65a WatchSource:0}: Error finding container 0186ae1be4a1224bc093f72e3346486b74302d4b7d41c2f061e2a7e8bb37d65a: Status 404 returned error can't find the container with id 0186ae1be4a1224bc093f72e3346486b74302d4b7d41c2f061e2a7e8bb37d65a Jul 30 00:15:02.611688 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:02.611647 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8667dbffc5-nm7rm" event={"ID":"d70e206a-eb89-4d5b-83e9-e08ddbdd8d35","Type":"ContainerStarted","Data":"0186ae1be4a1224bc093f72e3346486b74302d4b7d41c2f061e2a7e8bb37d65a"} Jul 30 00:15:04.620189 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:04.620153 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8667dbffc5-nm7rm" event={"ID":"d70e206a-eb89-4d5b-83e9-e08ddbdd8d35","Type":"ContainerStarted","Data":"e71f4120e9fcd4c2a1ddfb1e7c7e9ba629ef3c538df2cf00cc578338476c5a3e"} Jul 30 00:15:04.620641 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:04.620255 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8667dbffc5-nm7rm" Jul 30 00:15:04.637983 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:04.637935 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8667dbffc5-nm7rm" podStartSLOduration=2.366943532 podStartE2EDuration="4.637920972s" totalImagesPullingTime="2.27097744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:15:04.637307497 +0000 UTC m=+792.140565759" watchObservedRunningTime="2026-07-30 00:15:04.637920972 +0000 UTC m=+792.141179232" Jul 30 00:15:10.630118 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:10.630076 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8667dbffc5-nm7rm" Jul 30 00:15:11.608678 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:11.608649 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:15:11.700700 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:11.700662 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:15:24.607590 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:24.607562 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:15:40.093802 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.093768 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf"] Jul 30 00:15:40.099151 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.099131 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf" Jul 30 00:15:40.101511 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.101492 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 30 00:15:40.102540 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.102518 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dw4m6\"" Jul 30 00:15:40.102540 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.102530 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 30 00:15:40.102693 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.102546 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 30 00:15:40.106679 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.106657 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf"] Jul 30 00:15:40.134529 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.134505 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:15:40.240808 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.240780 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2a0a320-e247-4d6d-9230-de3819c88c54-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nbddf\" (UID: \"b2a0a320-e247-4d6d-9230-de3819c88c54\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf" Jul 30 00:15:40.240920 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.240822 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2a0a320-e247-4d6d-9230-de3819c88c54-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nbddf\" (UID: \"b2a0a320-e247-4d6d-9230-de3819c88c54\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf" Jul 30 00:15:40.240920 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.240843 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2a0a320-e247-4d6d-9230-de3819c88c54-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nbddf\" (UID: \"b2a0a320-e247-4d6d-9230-de3819c88c54\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf" Jul 30 00:15:40.240998 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.240916 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a0a320-e247-4d6d-9230-de3819c88c54-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nbddf\" (UID: \"b2a0a320-e247-4d6d-9230-de3819c88c54\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf" Jul 30 00:15:40.240998 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.240945 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx99k\" (UniqueName: \"kubernetes.io/projected/b2a0a320-e247-4d6d-9230-de3819c88c54-kube-api-access-nx99k\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nbddf\" (UID: \"b2a0a320-e247-4d6d-9230-de3819c88c54\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf" Jul 30 00:15:40.240998 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.240981 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2a0a320-e247-4d6d-9230-de3819c88c54-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nbddf\" (UID: \"b2a0a320-e247-4d6d-9230-de3819c88c54\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf" Jul 30 00:15:40.341828 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.341806 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2a0a320-e247-4d6d-9230-de3819c88c54-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nbddf\" (UID: \"b2a0a320-e247-4d6d-9230-de3819c88c54\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf" Jul 30 00:15:40.341952 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.341836 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2a0a320-e247-4d6d-9230-de3819c88c54-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nbddf\" (UID: \"b2a0a320-e247-4d6d-9230-de3819c88c54\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf" Jul 30 00:15:40.341952 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.341860 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a0a320-e247-4d6d-9230-de3819c88c54-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nbddf\" (UID: \"b2a0a320-e247-4d6d-9230-de3819c88c54\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf" Jul 30 00:15:40.341952 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.341880 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx99k\" (UniqueName: \"kubernetes.io/projected/b2a0a320-e247-4d6d-9230-de3819c88c54-kube-api-access-nx99k\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nbddf\" (UID: \"b2a0a320-e247-4d6d-9230-de3819c88c54\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf" Jul 30 00:15:40.341952 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.341918 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2a0a320-e247-4d6d-9230-de3819c88c54-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nbddf\" (UID: \"b2a0a320-e247-4d6d-9230-de3819c88c54\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf" Jul 30 00:15:40.341952 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.341941 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2a0a320-e247-4d6d-9230-de3819c88c54-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nbddf\" (UID: \"b2a0a320-e247-4d6d-9230-de3819c88c54\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf" Jul 30 00:15:40.342284 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.342261 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2a0a320-e247-4d6d-9230-de3819c88c54-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nbddf\" (UID: \"b2a0a320-e247-4d6d-9230-de3819c88c54\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf" Jul 30 00:15:40.342349 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.342283 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2a0a320-e247-4d6d-9230-de3819c88c54-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nbddf\" (UID: \"b2a0a320-e247-4d6d-9230-de3819c88c54\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf" Jul 30 00:15:40.342349 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.342333 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a0a320-e247-4d6d-9230-de3819c88c54-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nbddf\" (UID: \"b2a0a320-e247-4d6d-9230-de3819c88c54\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf" Jul 30 00:15:40.344197 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.344151 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2a0a320-e247-4d6d-9230-de3819c88c54-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nbddf\" (UID: \"b2a0a320-e247-4d6d-9230-de3819c88c54\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf" Jul 30 00:15:40.344342 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.344326 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2a0a320-e247-4d6d-9230-de3819c88c54-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nbddf\" (UID: \"b2a0a320-e247-4d6d-9230-de3819c88c54\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf" Jul 30 00:15:40.350074 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.350055 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx99k\" (UniqueName: \"kubernetes.io/projected/b2a0a320-e247-4d6d-9230-de3819c88c54-kube-api-access-nx99k\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nbddf\" (UID: \"b2a0a320-e247-4d6d-9230-de3819c88c54\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf" Jul 30 00:15:40.409819 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.409795 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf" Jul 30 00:15:40.530471 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.530449 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf"] Jul 30 00:15:40.532523 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:15:40.532498 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a0a320_e247_4d6d_9230_de3819c88c54.slice/crio-fb0f7ccac38f8b43fe20b38f8fd77867a3bbce95c63bee10f5220c1ef95d59d5 WatchSource:0}: Error finding container fb0f7ccac38f8b43fe20b38f8fd77867a3bbce95c63bee10f5220c1ef95d59d5: Status 404 returned error can't find the container with id fb0f7ccac38f8b43fe20b38f8fd77867a3bbce95c63bee10f5220c1ef95d59d5 Jul 30 00:15:40.751401 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:40.751317 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf" event={"ID":"b2a0a320-e247-4d6d-9230-de3819c88c54","Type":"ContainerStarted","Data":"fb0f7ccac38f8b43fe20b38f8fd77867a3bbce95c63bee10f5220c1ef95d59d5"} Jul 30 00:15:41.757094 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:41.757020 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf" event={"ID":"b2a0a320-e247-4d6d-9230-de3819c88c54","Type":"ContainerStarted","Data":"b070cc5b3bbbe3b986e24783c1c325b210759b1cbf08bd31337a83a082fd4a08"} Jul 30 00:15:41.774527 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:41.774480 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf" podStartSLOduration=0.850193652 podStartE2EDuration="1.774467073s" totalImagesPullingTime="924.273421ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:15:41.772757441 +0000 UTC m=+829.276015700" watchObservedRunningTime="2026-07-30 00:15:41.774467073 +0000 UTC m=+829.277725375" Jul 30 00:15:49.830518 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:49.830477 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:15:50.410978 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:50.410942 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf" Jul 30 00:15:50.411177 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:50.411028 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf" Jul 30 00:15:50.422984 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:50.422962 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf" Jul 30 00:15:50.800140 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:50.800058 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nbddf" Jul 30 00:15:57.920658 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:15:57.920626 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:16:02.397329 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:02.397295 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp"] Jul 30 00:16:02.400392 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:02.400376 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp" Jul 30 00:16:02.402949 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:02.402926 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 30 00:16:02.408622 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:02.408599 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp"] Jul 30 00:16:02.509919 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:02.509883 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/924002eb-17c0-426e-b65a-39519ffe69d5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5cpp\" (UID: \"924002eb-17c0-426e-b65a-39519ffe69d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp" Jul 30 00:16:02.509919 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:02.509918 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8pcm\" (UniqueName: \"kubernetes.io/projected/924002eb-17c0-426e-b65a-39519ffe69d5-kube-api-access-z8pcm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5cpp\" (UID: \"924002eb-17c0-426e-b65a-39519ffe69d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp" Jul 30 00:16:02.510117 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:02.509988 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/924002eb-17c0-426e-b65a-39519ffe69d5-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5cpp\" (UID: \"924002eb-17c0-426e-b65a-39519ffe69d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp" Jul 30 00:16:02.510117 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:02.510060 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/924002eb-17c0-426e-b65a-39519ffe69d5-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5cpp\" (UID: \"924002eb-17c0-426e-b65a-39519ffe69d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp" Jul 30 00:16:02.510117 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:02.510086 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/924002eb-17c0-426e-b65a-39519ffe69d5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5cpp\" (UID: \"924002eb-17c0-426e-b65a-39519ffe69d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp" Jul 30 00:16:02.510219 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:02.510152 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/924002eb-17c0-426e-b65a-39519ffe69d5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5cpp\" (UID: \"924002eb-17c0-426e-b65a-39519ffe69d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp" Jul 30 00:16:02.511886 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:02.511869 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:16:02.610876 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:02.610851 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/924002eb-17c0-426e-b65a-39519ffe69d5-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5cpp\" (UID: \"924002eb-17c0-426e-b65a-39519ffe69d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp" Jul 30 00:16:02.611014 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:02.610882 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/924002eb-17c0-426e-b65a-39519ffe69d5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5cpp\" (UID: \"924002eb-17c0-426e-b65a-39519ffe69d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp" Jul 30 00:16:02.611014 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:02.610899 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/924002eb-17c0-426e-b65a-39519ffe69d5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5cpp\" (UID: \"924002eb-17c0-426e-b65a-39519ffe69d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp" Jul 30 00:16:02.611139 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:02.611040 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/924002eb-17c0-426e-b65a-39519ffe69d5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5cpp\" (UID: \"924002eb-17c0-426e-b65a-39519ffe69d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp" Jul 30 00:16:02.611139 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:02.611085 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8pcm\" (UniqueName: \"kubernetes.io/projected/924002eb-17c0-426e-b65a-39519ffe69d5-kube-api-access-z8pcm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5cpp\" (UID: \"924002eb-17c0-426e-b65a-39519ffe69d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp" Jul 30 00:16:02.611305 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:02.611183 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/924002eb-17c0-426e-b65a-39519ffe69d5-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5cpp\" (UID: \"924002eb-17c0-426e-b65a-39519ffe69d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp" Jul 30 00:16:02.611355 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:02.611315 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/924002eb-17c0-426e-b65a-39519ffe69d5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5cpp\" (UID: \"924002eb-17c0-426e-b65a-39519ffe69d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp" Jul 30 00:16:02.611545 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:02.611517 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/924002eb-17c0-426e-b65a-39519ffe69d5-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5cpp\" (UID: \"924002eb-17c0-426e-b65a-39519ffe69d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp" Jul 30 00:16:02.611685 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:02.611581 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/924002eb-17c0-426e-b65a-39519ffe69d5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5cpp\" (UID: \"924002eb-17c0-426e-b65a-39519ffe69d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp" Jul 30 00:16:02.613485 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:02.613465 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/924002eb-17c0-426e-b65a-39519ffe69d5-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5cpp\" (UID: \"924002eb-17c0-426e-b65a-39519ffe69d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp" Jul 30 00:16:02.613669 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:02.613626 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/924002eb-17c0-426e-b65a-39519ffe69d5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5cpp\" (UID: \"924002eb-17c0-426e-b65a-39519ffe69d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp" Jul 30 00:16:02.618587 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:02.618563 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8pcm\" (UniqueName: \"kubernetes.io/projected/924002eb-17c0-426e-b65a-39519ffe69d5-kube-api-access-z8pcm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-d5cpp\" (UID: \"924002eb-17c0-426e-b65a-39519ffe69d5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp" Jul 30 00:16:02.710630 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:02.710578 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp" Jul 30 00:16:02.831168 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:02.831132 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp"] Jul 30 00:16:02.833894 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:16:02.833864 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924002eb_17c0_426e_b65a_39519ffe69d5.slice/crio-96580fde69f828ef0071d52bf50b34027fbf52ef34e0f740e6aad8af1232ac0e WatchSource:0}: Error finding container 96580fde69f828ef0071d52bf50b34027fbf52ef34e0f740e6aad8af1232ac0e: Status 404 returned error can't find the container with id 96580fde69f828ef0071d52bf50b34027fbf52ef34e0f740e6aad8af1232ac0e Jul 30 00:16:03.841743 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:03.841703 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp" event={"ID":"924002eb-17c0-426e-b65a-39519ffe69d5","Type":"ContainerStarted","Data":"3c4e3ee38b18bdc65785dcbf4446f9d902d720890209291b270dceedfef71827"} Jul 30 00:16:03.842134 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:03.841768 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp" event={"ID":"924002eb-17c0-426e-b65a-39519ffe69d5","Type":"ContainerStarted","Data":"96580fde69f828ef0071d52bf50b34027fbf52ef34e0f740e6aad8af1232ac0e"} Jul 30 00:16:03.862484 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:03.862435 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp" podStartSLOduration=1.684273182 podStartE2EDuration="1.862419727s" totalImagesPullingTime="178.146545ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:16:03.860967597 +0000 UTC m=+851.364226108" watchObservedRunningTime="2026-07-30 00:16:03.862419727 +0000 UTC m=+851.365677985" Jul 30 00:16:12.711552 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:12.711520 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp" Jul 30 00:16:12.712020 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:12.711590 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp" Jul 30 00:16:12.723211 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:12.723188 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp" Jul 30 00:16:12.890084 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:12.890065 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-d5cpp" Jul 30 00:16:21.637701 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:16:21.637668 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:17:09.771912 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:17:09.771873 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:17:11.649973 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:17:11.649942 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:17:38.931259 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:17:38.931224 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:17:59.351697 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:17:59.351650 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:17:59.447477 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:17:59.447447 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:18:00.470962 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:18:00.470933 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:18:00.569045 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:18:00.569018 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:19:12.507069 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:19:12.507018 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:20:05.017814 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:05.017725 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:20:06.295832 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:06.295795 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:20:06.417259 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:06.417226 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-76889fd47d-8tvfw"] Jul 30 00:20:06.420300 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:06.420277 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76889fd47d-8tvfw" Jul 30 00:20:06.424009 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:06.423985 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-j78v9\"" Jul 30 00:20:06.424118 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:06.423997 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 30 00:20:06.424118 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:06.424015 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 30 00:20:06.428990 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:06.428965 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76889fd47d-8tvfw"] Jul 30 00:20:06.467278 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:06.467255 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6dxb\" (UniqueName: \"kubernetes.io/projected/91baeab4-4ae3-428a-a5e8-96ff22f0773e-kube-api-access-g6dxb\") pod \"maas-controller-76889fd47d-8tvfw\" (UID: \"91baeab4-4ae3-428a-a5e8-96ff22f0773e\") " pod="opendatahub/maas-controller-76889fd47d-8tvfw" Jul 30 00:20:06.467386 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:06.467300 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/91baeab4-4ae3-428a-a5e8-96ff22f0773e-webhook-cert\") pod \"maas-controller-76889fd47d-8tvfw\" (UID: \"91baeab4-4ae3-428a-a5e8-96ff22f0773e\") " pod="opendatahub/maas-controller-76889fd47d-8tvfw" Jul 30 00:20:06.568304 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:06.568245 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6dxb\" (UniqueName: \"kubernetes.io/projected/91baeab4-4ae3-428a-a5e8-96ff22f0773e-kube-api-access-g6dxb\") pod \"maas-controller-76889fd47d-8tvfw\" (UID: \"91baeab4-4ae3-428a-a5e8-96ff22f0773e\") " pod="opendatahub/maas-controller-76889fd47d-8tvfw" Jul 30 00:20:06.568304 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:06.568297 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/91baeab4-4ae3-428a-a5e8-96ff22f0773e-webhook-cert\") pod \"maas-controller-76889fd47d-8tvfw\" (UID: \"91baeab4-4ae3-428a-a5e8-96ff22f0773e\") " pod="opendatahub/maas-controller-76889fd47d-8tvfw" Jul 30 00:20:06.570387 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:06.570370 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/91baeab4-4ae3-428a-a5e8-96ff22f0773e-webhook-cert\") pod \"maas-controller-76889fd47d-8tvfw\" (UID: \"91baeab4-4ae3-428a-a5e8-96ff22f0773e\") " pod="opendatahub/maas-controller-76889fd47d-8tvfw" Jul 30 00:20:06.577263 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:06.577244 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6dxb\" (UniqueName: \"kubernetes.io/projected/91baeab4-4ae3-428a-a5e8-96ff22f0773e-kube-api-access-g6dxb\") pod \"maas-controller-76889fd47d-8tvfw\" (UID: \"91baeab4-4ae3-428a-a5e8-96ff22f0773e\") " pod="opendatahub/maas-controller-76889fd47d-8tvfw" Jul 30 00:20:06.731701 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:06.731678 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76889fd47d-8tvfw" Jul 30 00:20:06.853029 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:06.852999 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76889fd47d-8tvfw"] Jul 30 00:20:06.856015 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:20:06.855988 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91baeab4_4ae3_428a_a5e8_96ff22f0773e.slice/crio-ef1f5ab55cecd4c71300b65c85b430b1c4b9bc97f8de07d62a44d6cd7bcca4ac WatchSource:0}: Error finding container ef1f5ab55cecd4c71300b65c85b430b1c4b9bc97f8de07d62a44d6cd7bcca4ac: Status 404 returned error can't find the container with id ef1f5ab55cecd4c71300b65c85b430b1c4b9bc97f8de07d62a44d6cd7bcca4ac Jul 30 00:20:06.857323 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:06.857305 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 00:20:07.783858 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:07.783818 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76889fd47d-8tvfw" event={"ID":"91baeab4-4ae3-428a-a5e8-96ff22f0773e","Type":"ContainerStarted","Data":"ef1f5ab55cecd4c71300b65c85b430b1c4b9bc97f8de07d62a44d6cd7bcca4ac"} Jul 30 00:20:08.789510 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:08.789471 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76889fd47d-8tvfw" event={"ID":"91baeab4-4ae3-428a-a5e8-96ff22f0773e","Type":"ContainerStarted","Data":"e42d5ab3e3a65535608488e265911f47bbcee8dafb77d21a329485c44dba8738"} Jul 30 00:20:08.789855 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:08.789605 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-76889fd47d-8tvfw" Jul 30 00:20:08.806739 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:08.806698 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-76889fd47d-8tvfw" podStartSLOduration=1.377744831 podStartE2EDuration="2.80668417s" totalImagesPullingTime="1.428939339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:20:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:20:08.80479914 +0000 UTC m=+1096.308057400" watchObservedRunningTime="2026-07-30 00:20:08.80668417 +0000 UTC m=+1096.309942430" Jul 30 00:20:19.795525 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:19.795494 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-76889fd47d-8tvfw" Jul 30 00:20:39.832358 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:39.832323 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:20:39.931323 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:39.931289 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:20:40.070927 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:40.070890 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:20:45.754894 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:45.754860 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:20:45.855960 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:45.855930 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:20:45.955038 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:45.955015 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:20:46.115872 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:46.115850 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:20:51.945872 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:51.945839 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:20:52.046501 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:20:52.046475 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:21:28.801317 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:21:28.801283 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:21:28.902006 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:21:28.901970 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:22:16.787080 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:22:16.787044 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:22:18.074866 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:22:18.074831 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:22:22.698774 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:22:22.698736 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:23:14.715361 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:23:14.715269 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:23:14.773464 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:23:14.773439 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:27:19.212292 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:27:19.212261 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:27:26.116695 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:27:26.116663 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:27:51.574534 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:27:51.574502 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:28:50.442240 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:50.442167 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:28:54.883854 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:54.883812 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn"] Jul 30 00:28:54.886870 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:54.886850 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" Jul 30 00:28:54.889370 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:54.889348 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-abb9f543-kserve-self-signed-certs\"" Jul 30 00:28:54.898796 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:54.898773 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn"] Jul 30 00:28:54.925309 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:54.925289 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:28:54.999749 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:54.999723 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff648796-5c28-457b-bc41-b0f0a949cf14-model-cache\") pod \"e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn\" (UID: \"ff648796-5c28-457b-bc41-b0f0a949cf14\") " pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" Jul 30 00:28:54.999868 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:54.999762 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff648796-5c28-457b-bc41-b0f0a949cf14-home\") pod \"e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn\" (UID: \"ff648796-5c28-457b-bc41-b0f0a949cf14\") " pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" Jul 30 00:28:54.999868 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:54.999815 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff648796-5c28-457b-bc41-b0f0a949cf14-dshm\") pod \"e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn\" (UID: \"ff648796-5c28-457b-bc41-b0f0a949cf14\") " pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" Jul 30 00:28:54.999868 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:54.999850 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff648796-5c28-457b-bc41-b0f0a949cf14-tmp-dir\") pod \"e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn\" (UID: \"ff648796-5c28-457b-bc41-b0f0a949cf14\") " pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" Jul 30 00:28:55.000021 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:54.999931 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff648796-5c28-457b-bc41-b0f0a949cf14-tls-certs\") pod \"e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn\" (UID: \"ff648796-5c28-457b-bc41-b0f0a949cf14\") " pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" Jul 30 00:28:55.000021 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:54.999968 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7xmw\" (UniqueName: \"kubernetes.io/projected/ff648796-5c28-457b-bc41-b0f0a949cf14-kube-api-access-h7xmw\") pod \"e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn\" (UID: \"ff648796-5c28-457b-bc41-b0f0a949cf14\") " pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" Jul 30 00:28:55.100632 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:55.100600 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff648796-5c28-457b-bc41-b0f0a949cf14-home\") pod \"e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn\" (UID: \"ff648796-5c28-457b-bc41-b0f0a949cf14\") " pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" Jul 30 00:28:55.100758 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:55.100644 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff648796-5c28-457b-bc41-b0f0a949cf14-dshm\") pod \"e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn\" (UID: \"ff648796-5c28-457b-bc41-b0f0a949cf14\") " pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" Jul 30 00:28:55.100798 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:55.100756 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff648796-5c28-457b-bc41-b0f0a949cf14-tmp-dir\") pod \"e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn\" (UID: \"ff648796-5c28-457b-bc41-b0f0a949cf14\") " pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" Jul 30 00:28:55.100798 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:55.100792 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff648796-5c28-457b-bc41-b0f0a949cf14-tls-certs\") pod \"e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn\" (UID: \"ff648796-5c28-457b-bc41-b0f0a949cf14\") " pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" Jul 30 00:28:55.100877 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:55.100821 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7xmw\" (UniqueName: \"kubernetes.io/projected/ff648796-5c28-457b-bc41-b0f0a949cf14-kube-api-access-h7xmw\") pod \"e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn\" (UID: \"ff648796-5c28-457b-bc41-b0f0a949cf14\") " pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" Jul 30 00:28:55.100914 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:55.100884 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff648796-5c28-457b-bc41-b0f0a949cf14-model-cache\") pod \"e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn\" (UID: \"ff648796-5c28-457b-bc41-b0f0a949cf14\") " pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" Jul 30 00:28:55.101012 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:55.100988 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff648796-5c28-457b-bc41-b0f0a949cf14-home\") pod \"e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn\" (UID: \"ff648796-5c28-457b-bc41-b0f0a949cf14\") " pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" Jul 30 00:28:55.101097 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:55.101077 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff648796-5c28-457b-bc41-b0f0a949cf14-tmp-dir\") pod \"e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn\" (UID: \"ff648796-5c28-457b-bc41-b0f0a949cf14\") " pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" Jul 30 00:28:55.101188 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:55.101174 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff648796-5c28-457b-bc41-b0f0a949cf14-model-cache\") pod \"e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn\" (UID: \"ff648796-5c28-457b-bc41-b0f0a949cf14\") " pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" Jul 30 00:28:55.102795 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:55.102774 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff648796-5c28-457b-bc41-b0f0a949cf14-dshm\") pod \"e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn\" (UID: \"ff648796-5c28-457b-bc41-b0f0a949cf14\") " pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" Jul 30 00:28:55.103070 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:55.103051 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff648796-5c28-457b-bc41-b0f0a949cf14-tls-certs\") pod \"e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn\" (UID: \"ff648796-5c28-457b-bc41-b0f0a949cf14\") " pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" Jul 30 00:28:55.109269 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:55.109249 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7xmw\" (UniqueName: \"kubernetes.io/projected/ff648796-5c28-457b-bc41-b0f0a949cf14-kube-api-access-h7xmw\") pod \"e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn\" (UID: \"ff648796-5c28-457b-bc41-b0f0a949cf14\") " pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" Jul 30 00:28:55.195971 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:55.195902 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" Jul 30 00:28:55.322325 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:55.322297 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn"] Jul 30 00:28:55.324907 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:28:55.324877 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff648796_5c28_457b_bc41_b0f0a949cf14.slice/crio-cefc22295be10b8d65cb37076ad451a36d5d5f8a025d3ab46eff89c457fb442d WatchSource:0}: Error finding container cefc22295be10b8d65cb37076ad451a36d5d5f8a025d3ab46eff89c457fb442d: Status 404 returned error can't find the container with id cefc22295be10b8d65cb37076ad451a36d5d5f8a025d3ab46eff89c457fb442d Jul 30 00:28:55.854297 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:55.854257 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" event={"ID":"ff648796-5c28-457b-bc41-b0f0a949cf14","Type":"ContainerStarted","Data":"df93ff2c8d046f18c90f36c23d017a942fe964a4d04dab26688b61eb35337404"} Jul 30 00:28:55.854297 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:55.854301 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" event={"ID":"ff648796-5c28-457b-bc41-b0f0a949cf14","Type":"ContainerStarted","Data":"cefc22295be10b8d65cb37076ad451a36d5d5f8a025d3ab46eff89c457fb442d"} Jul 30 00:28:55.870903 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:28:55.870858 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" podStartSLOduration=1.870843216 podStartE2EDuration="1.870843216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:28:55.869303637 +0000 UTC m=+1623.372561897" watchObservedRunningTime="2026-07-30 00:28:55.870843216 +0000 UTC m=+1623.374101482" Jul 30 00:29:05.197018 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:05.196978 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" Jul 30 00:29:05.197428 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:05.197038 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" Jul 30 00:29:05.209856 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:05.209828 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" Jul 30 00:29:05.908741 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:05.908714 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" Jul 30 00:29:26.119396 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:26.119363 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn"] Jul 30 00:29:27.159932 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:27.159903 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:29:27.215487 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:27.215458 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:29:41.138532 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:41.138457 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" podUID="ff648796-5c28-457b-bc41-b0f0a949cf14" containerName="main" containerID="cri-o://df93ff2c8d046f18c90f36c23d017a942fe964a4d04dab26688b61eb35337404" gracePeriod=45 Jul 30 00:29:41.378891 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:41.378868 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" Jul 30 00:29:41.474656 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:41.474587 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff648796-5c28-457b-bc41-b0f0a949cf14-model-cache\") pod \"ff648796-5c28-457b-bc41-b0f0a949cf14\" (UID: \"ff648796-5c28-457b-bc41-b0f0a949cf14\") " Jul 30 00:29:41.474656 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:41.474617 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7xmw\" (UniqueName: \"kubernetes.io/projected/ff648796-5c28-457b-bc41-b0f0a949cf14-kube-api-access-h7xmw\") pod \"ff648796-5c28-457b-bc41-b0f0a949cf14\" (UID: \"ff648796-5c28-457b-bc41-b0f0a949cf14\") " Jul 30 00:29:41.474853 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:41.474663 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff648796-5c28-457b-bc41-b0f0a949cf14-home\") pod \"ff648796-5c28-457b-bc41-b0f0a949cf14\" (UID: \"ff648796-5c28-457b-bc41-b0f0a949cf14\") " Jul 30 00:29:41.474853 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:41.474681 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff648796-5c28-457b-bc41-b0f0a949cf14-tls-certs\") pod \"ff648796-5c28-457b-bc41-b0f0a949cf14\" (UID: \"ff648796-5c28-457b-bc41-b0f0a949cf14\") " Jul 30 00:29:41.474853 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:41.474698 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff648796-5c28-457b-bc41-b0f0a949cf14-tmp-dir\") pod \"ff648796-5c28-457b-bc41-b0f0a949cf14\" (UID: \"ff648796-5c28-457b-bc41-b0f0a949cf14\") " Jul 30 00:29:41.474853 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:41.474718 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff648796-5c28-457b-bc41-b0f0a949cf14-dshm\") pod \"ff648796-5c28-457b-bc41-b0f0a949cf14\" (UID: \"ff648796-5c28-457b-bc41-b0f0a949cf14\") " Jul 30 00:29:41.475021 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:41.474877 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff648796-5c28-457b-bc41-b0f0a949cf14-model-cache" (OuterVolumeSpecName: "model-cache") pod "ff648796-5c28-457b-bc41-b0f0a949cf14" (UID: "ff648796-5c28-457b-bc41-b0f0a949cf14"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:29:41.475021 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:41.474947 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff648796-5c28-457b-bc41-b0f0a949cf14-home" (OuterVolumeSpecName: "home") pod "ff648796-5c28-457b-bc41-b0f0a949cf14" (UID: "ff648796-5c28-457b-bc41-b0f0a949cf14"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:29:41.475021 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:41.474961 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff648796-5c28-457b-bc41-b0f0a949cf14-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ff648796-5c28-457b-bc41-b0f0a949cf14" (UID: "ff648796-5c28-457b-bc41-b0f0a949cf14"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:29:41.476786 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:41.476760 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff648796-5c28-457b-bc41-b0f0a949cf14-dshm" (OuterVolumeSpecName: "dshm") pod "ff648796-5c28-457b-bc41-b0f0a949cf14" (UID: "ff648796-5c28-457b-bc41-b0f0a949cf14"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:29:41.476895 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:41.476764 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff648796-5c28-457b-bc41-b0f0a949cf14-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ff648796-5c28-457b-bc41-b0f0a949cf14" (UID: "ff648796-5c28-457b-bc41-b0f0a949cf14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 00:29:41.476895 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:41.476815 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff648796-5c28-457b-bc41-b0f0a949cf14-kube-api-access-h7xmw" (OuterVolumeSpecName: "kube-api-access-h7xmw") pod "ff648796-5c28-457b-bc41-b0f0a949cf14" (UID: "ff648796-5c28-457b-bc41-b0f0a949cf14"). InnerVolumeSpecName "kube-api-access-h7xmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:29:41.575481 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:41.575454 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff648796-5c28-457b-bc41-b0f0a949cf14-model-cache\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:29:41.575481 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:41.575474 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7xmw\" (UniqueName: \"kubernetes.io/projected/ff648796-5c28-457b-bc41-b0f0a949cf14-kube-api-access-h7xmw\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:29:41.575481 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:41.575484 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff648796-5c28-457b-bc41-b0f0a949cf14-home\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:29:41.575646 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:41.575494 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff648796-5c28-457b-bc41-b0f0a949cf14-tls-certs\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:29:41.575646 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:41.575503 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff648796-5c28-457b-bc41-b0f0a949cf14-tmp-dir\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:29:41.575646 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:41.575510 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff648796-5c28-457b-bc41-b0f0a949cf14-dshm\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:29:42.040979 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:42.040943 2572 generic.go:356] "Generic (PLEG): container finished" podID="ff648796-5c28-457b-bc41-b0f0a949cf14" containerID="df93ff2c8d046f18c90f36c23d017a942fe964a4d04dab26688b61eb35337404" exitCode=0 Jul 30 00:29:42.041180 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:42.040986 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" Jul 30 00:29:42.041180 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:42.041029 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" event={"ID":"ff648796-5c28-457b-bc41-b0f0a949cf14","Type":"ContainerDied","Data":"df93ff2c8d046f18c90f36c23d017a942fe964a4d04dab26688b61eb35337404"} Jul 30 00:29:42.041180 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:42.041076 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn" event={"ID":"ff648796-5c28-457b-bc41-b0f0a949cf14","Type":"ContainerDied","Data":"cefc22295be10b8d65cb37076ad451a36d5d5f8a025d3ab46eff89c457fb442d"} Jul 30 00:29:42.041180 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:42.041101 2572 scope.go:117] "RemoveContainer" containerID="df93ff2c8d046f18c90f36c23d017a942fe964a4d04dab26688b61eb35337404" Jul 30 00:29:42.050414 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:42.050395 2572 scope.go:117] "RemoveContainer" containerID="df93ff2c8d046f18c90f36c23d017a942fe964a4d04dab26688b61eb35337404" Jul 30 00:29:42.050664 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:29:42.050643 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df93ff2c8d046f18c90f36c23d017a942fe964a4d04dab26688b61eb35337404\": container with ID starting with df93ff2c8d046f18c90f36c23d017a942fe964a4d04dab26688b61eb35337404 not found: ID does not exist" containerID="df93ff2c8d046f18c90f36c23d017a942fe964a4d04dab26688b61eb35337404" Jul 30 00:29:42.050736 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:42.050675 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df93ff2c8d046f18c90f36c23d017a942fe964a4d04dab26688b61eb35337404"} err="failed to get container status \"df93ff2c8d046f18c90f36c23d017a942fe964a4d04dab26688b61eb35337404\": rpc error: code = NotFound desc = could not find container \"df93ff2c8d046f18c90f36c23d017a942fe964a4d04dab26688b61eb35337404\": container with ID starting with df93ff2c8d046f18c90f36c23d017a942fe964a4d04dab26688b61eb35337404 not found: ID does not exist" Jul 30 00:29:42.061910 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:42.061883 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn"] Jul 30 00:29:42.066836 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:42.066817 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-abb9f543-kserve-6bc7cb7567-kgbbn"] Jul 30 00:29:42.993317 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:29:42.993283 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff648796-5c28-457b-bc41-b0f0a949cf14" path="/var/lib/kubelet/pods/ff648796-5c28-457b-bc41-b0f0a949cf14/volumes" Jul 30 00:32:07.834365 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:32:07.834331 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:32:36.114214 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:32:36.114176 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:32:57.629886 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:32:57.629848 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:00.619806 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:00.619745 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:00.714874 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:00.714833 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:01.033098 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:01.033008 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:06.888812 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:06.888774 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:06.947032 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:06.947003 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:07.112388 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.112353 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:07.385521 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.385490 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b"] Jul 30 00:33:07.385787 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.385775 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff648796-5c28-457b-bc41-b0f0a949cf14" containerName="main" Jul 30 00:33:07.385829 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.385789 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff648796-5c28-457b-bc41-b0f0a949cf14" containerName="main" Jul 30 00:33:07.385862 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.385843 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff648796-5c28-457b-bc41-b0f0a949cf14" containerName="main" Jul 30 00:33:07.388762 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.388746 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.391540 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.391519 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9092d698-openshift-default-dockercfg-g99ss\"" Jul 30 00:33:07.409782 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.409756 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b"] Jul 30 00:33:07.412842 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.412818 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:07.504731 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.504702 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-credential-socket\") pod \"e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.504895 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.504736 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-workload-socket\") pod \"e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.504895 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.504770 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/691163c3-96d1-4a1b-b19a-253e164f943c-istio-podinfo\") pod \"e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.504895 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.504843 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-workload-certs\") pod \"e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.504895 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.504870 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/691163c3-96d1-4a1b-b19a-253e164f943c-istio-token\") pod \"e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.504895 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.504889 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/691163c3-96d1-4a1b-b19a-253e164f943c-istiod-ca-cert\") pod \"e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.505053 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.504907 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m27h\" (UniqueName: \"kubernetes.io/projected/691163c3-96d1-4a1b-b19a-253e164f943c-kube-api-access-8m27h\") pod \"e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.505053 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.504943 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-istio-envoy\") pod \"e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.505053 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.504972 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-istio-data\") pod \"e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.556056 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.556022 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:07.605854 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.605809 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-credential-socket\") pod \"e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.605854 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.605862 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-workload-socket\") pod \"e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.606077 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.605893 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/691163c3-96d1-4a1b-b19a-253e164f943c-istio-podinfo\") pod \"e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.606077 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.605937 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-workload-certs\") pod \"e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.606077 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.605956 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/691163c3-96d1-4a1b-b19a-253e164f943c-istio-token\") pod \"e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.606077 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.605974 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/691163c3-96d1-4a1b-b19a-253e164f943c-istiod-ca-cert\") pod \"e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.606077 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.605993 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m27h\" (UniqueName: \"kubernetes.io/projected/691163c3-96d1-4a1b-b19a-253e164f943c-kube-api-access-8m27h\") pod \"e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.606077 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.606023 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-istio-envoy\") pod \"e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.606077 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.606060 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-istio-data\") pod \"e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.606491 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.606292 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-credential-socket\") pod \"e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.606491 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.606314 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-workload-socket\") pod \"e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.606491 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.606376 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-workload-certs\") pod \"e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.606491 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.606406 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-istio-data\") pod \"e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.606736 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.606718 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/691163c3-96d1-4a1b-b19a-253e164f943c-istiod-ca-cert\") pod \"e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.608298 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.608276 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-istio-envoy\") pod \"e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.608402 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.608332 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/691163c3-96d1-4a1b-b19a-253e164f943c-istio-podinfo\") pod \"e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.614510 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.614487 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/691163c3-96d1-4a1b-b19a-253e164f943c-istio-token\") pod \"e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.614659 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.614638 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m27h\" (UniqueName: \"kubernetes.io/projected/691163c3-96d1-4a1b-b19a-253e164f943c-kube-api-access-8m27h\") pod \"e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.699432 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.699334 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:07.832239 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.832177 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b"] Jul 30 00:33:07.834698 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:33:07.834666 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod691163c3_96d1_4a1b_b19a_253e164f943c.slice/crio-52221d7eeab496c59e0920829525dee34aa8a271842b59a6a18d22b4d4326802 WatchSource:0}: Error finding container 52221d7eeab496c59e0920829525dee34aa8a271842b59a6a18d22b4d4326802: Status 404 returned error can't find the container with id 52221d7eeab496c59e0920829525dee34aa8a271842b59a6a18d22b4d4326802 Jul 30 00:33:07.837273 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.837240 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:33:07.837347 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.837305 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:33:07.837347 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.837337 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:33:07.858526 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:07.858498 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" event={"ID":"691163c3-96d1-4a1b-b19a-253e164f943c","Type":"ContainerStarted","Data":"52221d7eeab496c59e0920829525dee34aa8a271842b59a6a18d22b4d4326802"} Jul 30 00:33:08.864337 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:08.864302 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" event={"ID":"691163c3-96d1-4a1b-b19a-253e164f943c","Type":"ContainerStarted","Data":"8c2ebd8b78cc3496ab7caa2810a70f0a4aed208768c41c548792d9dfca0f1f75"} Jul 30 00:33:08.885779 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:08.885732 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" podStartSLOduration=1.885715319 podStartE2EDuration="1.885715319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:33:08.883583304 +0000 UTC m=+1876.386841565" watchObservedRunningTime="2026-07-30 00:33:08.885715319 +0000 UTC m=+1876.388973640" Jul 30 00:33:09.700472 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:09.700435 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:09.705328 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:09.705297 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:09.868937 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:09.868909 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:09.870003 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:09.869986 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:10.217291 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:10.217258 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:10.312461 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:10.312421 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:11.338905 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:11.338872 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:11.441289 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:11.441251 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:11.959152 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:11.959100 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:13.647773 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:13.647730 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:25.499721 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:25.499655 2572 patch_prober.go:28] interesting pod/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": read tcp 10.133.0.2:52876->10.133.0.23:15021: read: connection reset by peer" start-of-body= Jul 30 00:33:25.500089 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:25.499729 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" podUID="691163c3-96d1-4a1b-b19a-253e164f943c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": read tcp 10.133.0.2:52876->10.133.0.23:15021: read: connection reset by peer" Jul 30 00:33:25.933306 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:25.933271 2572 generic.go:356] "Generic (PLEG): container finished" podID="691163c3-96d1-4a1b-b19a-253e164f943c" containerID="8c2ebd8b78cc3496ab7caa2810a70f0a4aed208768c41c548792d9dfca0f1f75" exitCode=0 Jul 30 00:33:25.933470 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:25.933345 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" event={"ID":"691163c3-96d1-4a1b-b19a-253e164f943c","Type":"ContainerDied","Data":"8c2ebd8b78cc3496ab7caa2810a70f0a4aed208768c41c548792d9dfca0f1f75"} Jul 30 00:33:25.933715 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:25.933701 2572 scope.go:117] "RemoveContainer" containerID="8c2ebd8b78cc3496ab7caa2810a70f0a4aed208768c41c548792d9dfca0f1f75" Jul 30 00:33:25.934671 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:25.934647 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:33:25.934722 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:25.934710 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:33:25.934756 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:25.934744 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:33:26.942226 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:26.942185 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" event={"ID":"691163c3-96d1-4a1b-b19a-253e164f943c","Type":"ContainerStarted","Data":"078e812cbdb59b8b3186b3b761fe4d3b1c5607d827cf61e9670e8aa23ca2f14e"} Jul 30 00:33:27.700168 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:27.700127 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:27.700382 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:27.700340 2572 patch_prober.go:28] interesting pod/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Jul 30 00:33:27.700424 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:27.700398 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" podUID="691163c3-96d1-4a1b-b19a-253e164f943c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Jul 30 00:33:28.699712 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:28.699682 2572 patch_prober.go:28] interesting pod/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Jul 30 00:33:28.700167 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:28.699747 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" podUID="691163c3-96d1-4a1b-b19a-253e164f943c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Jul 30 00:33:29.700576 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:29.700535 2572 patch_prober.go:28] interesting pod/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Jul 30 00:33:29.700971 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:29.700615 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" podUID="691163c3-96d1-4a1b-b19a-253e164f943c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Jul 30 00:33:30.700309 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:30.700274 2572 patch_prober.go:28] interesting pod/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Jul 30 00:33:30.700482 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:30.700341 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" podUID="691163c3-96d1-4a1b-b19a-253e164f943c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Jul 30 00:33:31.703568 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:31.703536 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:31.703948 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:31.703786 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:31.704347 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:31.704328 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:41.753311 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:41.753275 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:41.853999 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:41.853953 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:41.916787 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:41.916748 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:41.970142 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:41.970100 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:42.040629 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:42.040547 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b"] Jul 30 00:33:42.040786 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:42.040766 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" podUID="691163c3-96d1-4a1b-b19a-253e164f943c" containerName="istio-proxy" containerID="cri-o://078e812cbdb59b8b3186b3b761fe4d3b1c5607d827cf61e9670e8aa23ca2f14e" gracePeriod=30 Jul 30 00:33:42.068338 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:42.068309 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:42.286434 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:42.286395 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:42.541508 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:42.541471 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:42.612040 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:42.612001 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:42.741210 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:42.741177 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:43.011829 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.011795 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2"] Jul 30 00:33:43.014315 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.014296 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.016788 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.016768 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-e5dec626-gw-openshift-default-dockercfg-rkzzh\"" Jul 30 00:33:43.026266 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.026240 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2"] Jul 30 00:33:43.040315 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.040294 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:43.135983 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.135950 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-workload-certs\") pod \"e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.136165 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.135989 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istio-data\") pod \"e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.136165 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.136014 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istio-podinfo\") pod \"e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.136165 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.136033 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd8v5\" (UniqueName: \"kubernetes.io/projected/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-kube-api-access-hd8v5\") pod \"e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.136165 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.136057 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-credential-socket\") pod \"e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.136165 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.136075 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-workload-socket\") pod \"e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.136165 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.136153 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istio-token\") pod \"e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.136369 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.136170 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istiod-ca-cert\") pod \"e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.136369 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.136200 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istio-envoy\") pod \"e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.237192 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.237156 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-credential-socket\") pod \"e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.237192 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.237196 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-workload-socket\") pod \"e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.237389 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.237230 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istio-token\") pod \"e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.237389 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.237252 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istiod-ca-cert\") pod \"e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.237389 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.237283 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istio-envoy\") pod \"e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.237389 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.237346 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-workload-certs\") pod \"e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.237389 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.237382 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istio-data\") pod \"e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.237625 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.237422 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istio-podinfo\") pod \"e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.237625 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.237446 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd8v5\" (UniqueName: \"kubernetes.io/projected/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-kube-api-access-hd8v5\") pod \"e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.237730 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.237664 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-workload-socket\") pod \"e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.237816 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.237795 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-workload-certs\") pod \"e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.237874 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.237854 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-credential-socket\") pod \"e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.237922 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.237883 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istio-data\") pod \"e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.238121 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.238094 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istiod-ca-cert\") pod \"e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.239568 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.239546 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istio-envoy\") pod \"e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.239809 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.239788 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istio-podinfo\") pod \"e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.246363 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.246340 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istio-token\") pod \"e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.246464 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.246358 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd8v5\" (UniqueName: \"kubernetes.io/projected/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-kube-api-access-hd8v5\") pod \"e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.297902 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.297812 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:43.323579 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.323551 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:43.455590 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.455565 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2"] Jul 30 00:33:43.457525 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:33:43.457493 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92b1109_04cb_4ad8_87c4_27c6b00c7a13.slice/crio-dd613df708bfa74e6f09c3b3619c58612e6dc1df8725a42e291f154850cfd380 WatchSource:0}: Error finding container dd613df708bfa74e6f09c3b3619c58612e6dc1df8725a42e291f154850cfd380: Status 404 returned error can't find the container with id dd613df708bfa74e6f09c3b3619c58612e6dc1df8725a42e291f154850cfd380 Jul 30 00:33:43.459798 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.459762 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:33:43.459880 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.459844 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:33:43.459919 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.459882 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:33:43.516527 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:43.516501 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:44.014176 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:44.014138 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" event={"ID":"e92b1109-04cb-4ad8-87c4-27c6b00c7a13","Type":"ContainerStarted","Data":"2a59318837be5c6cb7053fb62a956cdbac199a636d8fc8a7055fc719e956ad92"} Jul 30 00:33:44.014549 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:44.014204 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" event={"ID":"e92b1109-04cb-4ad8-87c4-27c6b00c7a13","Type":"ContainerStarted","Data":"dd613df708bfa74e6f09c3b3619c58612e6dc1df8725a42e291f154850cfd380"} Jul 30 00:33:44.034301 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:44.034245 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" podStartSLOduration=2.03422737 podStartE2EDuration="2.03422737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:33:44.032995609 +0000 UTC m=+1911.536253873" watchObservedRunningTime="2026-07-30 00:33:44.03422737 +0000 UTC m=+1911.537485630" Jul 30 00:33:44.324681 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:44.324594 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:44.329195 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:44.329171 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:45.023989 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:45.023958 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:45.025041 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:45.025018 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:33:45.471027 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:45.470981 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:45.795901 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:45.795817 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:45.953579 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:45.953548 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:46.218812 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:46.218775 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:46.308406 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:46.308377 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:33:46.706907 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:46.706871 2572 patch_prober.go:28] interesting pod/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 30 00:33:46.707067 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:46.706924 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" podUID="691163c3-96d1-4a1b-b19a-253e164f943c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 30 00:33:47.293193 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:47.293169 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:47.478029 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:47.477994 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/691163c3-96d1-4a1b-b19a-253e164f943c-istio-token\") pod \"691163c3-96d1-4a1b-b19a-253e164f943c\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " Jul 30 00:33:47.478243 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:47.478045 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-workload-certs\") pod \"691163c3-96d1-4a1b-b19a-253e164f943c\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " Jul 30 00:33:47.478243 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:47.478082 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/691163c3-96d1-4a1b-b19a-253e164f943c-istio-podinfo\") pod \"691163c3-96d1-4a1b-b19a-253e164f943c\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " Jul 30 00:33:47.478243 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:47.478101 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-workload-socket\") pod \"691163c3-96d1-4a1b-b19a-253e164f943c\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " Jul 30 00:33:47.478243 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:47.478159 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/691163c3-96d1-4a1b-b19a-253e164f943c-istiod-ca-cert\") pod \"691163c3-96d1-4a1b-b19a-253e164f943c\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " Jul 30 00:33:47.478549 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:47.478354 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-istio-envoy\") pod \"691163c3-96d1-4a1b-b19a-253e164f943c\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " Jul 30 00:33:47.478549 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:47.478414 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "691163c3-96d1-4a1b-b19a-253e164f943c" (UID: "691163c3-96d1-4a1b-b19a-253e164f943c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:33:47.478549 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:47.478434 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-credential-socket\") pod \"691163c3-96d1-4a1b-b19a-253e164f943c\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " Jul 30 00:33:47.478549 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:47.478440 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "691163c3-96d1-4a1b-b19a-253e164f943c" (UID: "691163c3-96d1-4a1b-b19a-253e164f943c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:33:47.478549 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:47.478477 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8m27h\" (UniqueName: \"kubernetes.io/projected/691163c3-96d1-4a1b-b19a-253e164f943c-kube-api-access-8m27h\") pod \"691163c3-96d1-4a1b-b19a-253e164f943c\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " Jul 30 00:33:47.478549 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:47.478503 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-istio-data\") pod \"691163c3-96d1-4a1b-b19a-253e164f943c\" (UID: \"691163c3-96d1-4a1b-b19a-253e164f943c\") " Jul 30 00:33:47.478549 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:47.478508 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/691163c3-96d1-4a1b-b19a-253e164f943c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "691163c3-96d1-4a1b-b19a-253e164f943c" (UID: "691163c3-96d1-4a1b-b19a-253e164f943c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:33:47.478907 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:47.478808 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-workload-socket\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:33:47.478907 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:47.478829 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/691163c3-96d1-4a1b-b19a-253e164f943c-istiod-ca-cert\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:33:47.478907 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:47.478826 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "691163c3-96d1-4a1b-b19a-253e164f943c" (UID: "691163c3-96d1-4a1b-b19a-253e164f943c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:33:47.478907 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:47.478843 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-workload-certs\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:33:47.479592 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:47.479563 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-istio-data" (OuterVolumeSpecName: "istio-data") pod "691163c3-96d1-4a1b-b19a-253e164f943c" (UID: "691163c3-96d1-4a1b-b19a-253e164f943c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:33:47.480461 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:47.480439 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/691163c3-96d1-4a1b-b19a-253e164f943c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "691163c3-96d1-4a1b-b19a-253e164f943c" (UID: "691163c3-96d1-4a1b-b19a-253e164f943c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 00:33:47.480461 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:47.480449 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/691163c3-96d1-4a1b-b19a-253e164f943c-istio-token" (OuterVolumeSpecName: "istio-token") pod "691163c3-96d1-4a1b-b19a-253e164f943c" (UID: "691163c3-96d1-4a1b-b19a-253e164f943c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:33:47.480620 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:47.480539 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "691163c3-96d1-4a1b-b19a-253e164f943c" (UID: "691163c3-96d1-4a1b-b19a-253e164f943c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:33:47.480698 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:47.480679 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/691163c3-96d1-4a1b-b19a-253e164f943c-kube-api-access-8m27h" (OuterVolumeSpecName: "kube-api-access-8m27h") pod "691163c3-96d1-4a1b-b19a-253e164f943c" (UID: "691163c3-96d1-4a1b-b19a-253e164f943c"). InnerVolumeSpecName "kube-api-access-8m27h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:33:47.579429 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:47.579392 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-istio-envoy\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:33:47.579429 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:47.579423 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-credential-socket\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:33:47.579429 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:47.579436 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m27h\" (UniqueName: \"kubernetes.io/projected/691163c3-96d1-4a1b-b19a-253e164f943c-kube-api-access-8m27h\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:33:47.579656 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:47.579450 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/691163c3-96d1-4a1b-b19a-253e164f943c-istio-data\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:33:47.579656 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:47.579462 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/691163c3-96d1-4a1b-b19a-253e164f943c-istio-token\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:33:47.579656 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:47.579472 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/691163c3-96d1-4a1b-b19a-253e164f943c-istio-podinfo\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:33:48.042140 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:48.042082 2572 generic.go:356] "Generic (PLEG): container finished" podID="691163c3-96d1-4a1b-b19a-253e164f943c" containerID="078e812cbdb59b8b3186b3b761fe4d3b1c5607d827cf61e9670e8aa23ca2f14e" exitCode=0 Jul 30 00:33:48.042313 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:48.042139 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" event={"ID":"691163c3-96d1-4a1b-b19a-253e164f943c","Type":"ContainerDied","Data":"078e812cbdb59b8b3186b3b761fe4d3b1c5607d827cf61e9670e8aa23ca2f14e"} Jul 30 00:33:48.042313 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:48.042174 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" Jul 30 00:33:48.042313 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:48.042199 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b" event={"ID":"691163c3-96d1-4a1b-b19a-253e164f943c","Type":"ContainerDied","Data":"52221d7eeab496c59e0920829525dee34aa8a271842b59a6a18d22b4d4326802"} Jul 30 00:33:48.042313 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:48.042226 2572 scope.go:117] "RemoveContainer" containerID="078e812cbdb59b8b3186b3b761fe4d3b1c5607d827cf61e9670e8aa23ca2f14e" Jul 30 00:33:48.050913 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:48.050895 2572 scope.go:117] "RemoveContainer" containerID="8c2ebd8b78cc3496ab7caa2810a70f0a4aed208768c41c548792d9dfca0f1f75" Jul 30 00:33:48.058764 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:48.058748 2572 scope.go:117] "RemoveContainer" containerID="078e812cbdb59b8b3186b3b761fe4d3b1c5607d827cf61e9670e8aa23ca2f14e" Jul 30 00:33:48.059015 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:33:48.058995 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"078e812cbdb59b8b3186b3b761fe4d3b1c5607d827cf61e9670e8aa23ca2f14e\": container with ID starting with 078e812cbdb59b8b3186b3b761fe4d3b1c5607d827cf61e9670e8aa23ca2f14e not found: ID does not exist" containerID="078e812cbdb59b8b3186b3b761fe4d3b1c5607d827cf61e9670e8aa23ca2f14e" Jul 30 00:33:48.059060 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:48.059023 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"078e812cbdb59b8b3186b3b761fe4d3b1c5607d827cf61e9670e8aa23ca2f14e"} err="failed to get container status \"078e812cbdb59b8b3186b3b761fe4d3b1c5607d827cf61e9670e8aa23ca2f14e\": rpc error: code = NotFound desc = could not find container \"078e812cbdb59b8b3186b3b761fe4d3b1c5607d827cf61e9670e8aa23ca2f14e\": container with ID starting with 078e812cbdb59b8b3186b3b761fe4d3b1c5607d827cf61e9670e8aa23ca2f14e not found: ID does not exist" Jul 30 00:33:48.059060 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:48.059041 2572 scope.go:117] "RemoveContainer" containerID="8c2ebd8b78cc3496ab7caa2810a70f0a4aed208768c41c548792d9dfca0f1f75" Jul 30 00:33:48.059664 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:33:48.059642 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c2ebd8b78cc3496ab7caa2810a70f0a4aed208768c41c548792d9dfca0f1f75\": container with ID starting with 8c2ebd8b78cc3496ab7caa2810a70f0a4aed208768c41c548792d9dfca0f1f75 not found: ID does not exist" containerID="8c2ebd8b78cc3496ab7caa2810a70f0a4aed208768c41c548792d9dfca0f1f75" Jul 30 00:33:48.059751 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:48.059671 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c2ebd8b78cc3496ab7caa2810a70f0a4aed208768c41c548792d9dfca0f1f75"} err="failed to get container status \"8c2ebd8b78cc3496ab7caa2810a70f0a4aed208768c41c548792d9dfca0f1f75\": rpc error: code = NotFound desc = could not find container \"8c2ebd8b78cc3496ab7caa2810a70f0a4aed208768c41c548792d9dfca0f1f75\": container with ID starting with 8c2ebd8b78cc3496ab7caa2810a70f0a4aed208768c41c548792d9dfca0f1f75 not found: ID does not exist" Jul 30 00:33:48.067020 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:48.066998 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b"] Jul 30 00:33:48.070720 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:48.070699 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9092d698-openshift-default-7df756b78b-kxr9b"] Jul 30 00:33:48.992192 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:33:48.992149 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="691163c3-96d1-4a1b-b19a-253e164f943c" path="/var/lib/kubelet/pods/691163c3-96d1-4a1b-b19a-253e164f943c/volumes" Jul 30 00:34:00.025992 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:00.025907 2572 patch_prober.go:28] interesting pod/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Jul 30 00:34:00.026386 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:00.025986 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" podUID="e92b1109-04cb-4ad8-87c4-27c6b00c7a13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Jul 30 00:34:00.093119 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:00.093016 2572 generic.go:356] "Generic (PLEG): container finished" podID="e92b1109-04cb-4ad8-87c4-27c6b00c7a13" containerID="2a59318837be5c6cb7053fb62a956cdbac199a636d8fc8a7055fc719e956ad92" exitCode=0 Jul 30 00:34:00.093299 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:00.093093 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" event={"ID":"e92b1109-04cb-4ad8-87c4-27c6b00c7a13","Type":"ContainerDied","Data":"2a59318837be5c6cb7053fb62a956cdbac199a636d8fc8a7055fc719e956ad92"} Jul 30 00:34:00.093551 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:00.093533 2572 scope.go:117] "RemoveContainer" containerID="2a59318837be5c6cb7053fb62a956cdbac199a636d8fc8a7055fc719e956ad92" Jul 30 00:34:00.094544 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:00.094518 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:34:00.094604 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:00.094592 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:34:00.094655 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:00.094642 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:34:01.099087 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:01.099055 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" event={"ID":"e92b1109-04cb-4ad8-87c4-27c6b00c7a13","Type":"ContainerStarted","Data":"a6a4ed923e0758abce9c6a2dfa86baca47630a3d94f8599d8256527f77ae74e9"} Jul 30 00:34:01.324634 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:01.324589 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:34:01.324862 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:01.324835 2572 patch_prober.go:28] interesting pod/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Jul 30 00:34:01.324913 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:01.324890 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" podUID="e92b1109-04cb-4ad8-87c4-27c6b00c7a13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Jul 30 00:34:02.324875 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:02.324837 2572 patch_prober.go:28] interesting pod/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Jul 30 00:34:02.325307 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:02.324905 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" podUID="e92b1109-04cb-4ad8-87c4-27c6b00c7a13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Jul 30 00:34:03.324794 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:03.324757 2572 patch_prober.go:28] interesting pod/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Jul 30 00:34:03.324980 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:03.324812 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" podUID="e92b1109-04cb-4ad8-87c4-27c6b00c7a13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Jul 30 00:34:04.324175 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:04.324137 2572 patch_prober.go:28] interesting pod/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Jul 30 00:34:04.324386 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:04.324201 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" podUID="e92b1109-04cb-4ad8-87c4-27c6b00c7a13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Jul 30 00:34:05.328175 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:05.328147 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:34:05.328599 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:05.328415 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:34:05.329167 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:05.329147 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:34:16.289657 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:16.289625 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:16.386889 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:16.386845 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:16.445704 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:16.445671 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:16.505769 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:16.505742 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:16.592923 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:16.586699 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2"] Jul 30 00:34:16.592923 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:16.586971 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" podUID="e92b1109-04cb-4ad8-87c4-27c6b00c7a13" containerName="istio-proxy" containerID="cri-o://a6a4ed923e0758abce9c6a2dfa86baca47630a3d94f8599d8256527f77ae74e9" gracePeriod=30 Jul 30 00:34:16.609419 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:16.607656 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:16.834443 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:16.834410 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:17.066388 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:17.066306 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:17.361214 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:17.361181 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:17.634317 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:17.634221 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:20.158806 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:20.158775 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:20.257086 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:20.257051 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:20.330874 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:20.330837 2572 patch_prober.go:28] interesting pod/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 30 00:34:20.331023 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:20.330890 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" podUID="e92b1109-04cb-4ad8-87c4-27c6b00c7a13" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 30 00:34:21.249592 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.249562 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cd573186-549f84cd45-28nwz"] Jul 30 00:34:21.249942 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.249852 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="691163c3-96d1-4a1b-b19a-253e164f943c" containerName="istio-proxy" Jul 30 00:34:21.249942 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.249862 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="691163c3-96d1-4a1b-b19a-253e164f943c" containerName="istio-proxy" Jul 30 00:34:21.249942 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.249934 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="691163c3-96d1-4a1b-b19a-253e164f943c" containerName="istio-proxy" Jul 30 00:34:21.252905 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.252885 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd573186-549f84cd45-28nwz" Jul 30 00:34:21.255403 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.255379 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-cd573186-serving-cert\"" Jul 30 00:34:21.264118 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.264002 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cd573186-549f84cd45-28nwz"] Jul 30 00:34:21.289543 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.289510 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:21.388458 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.388425 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a90ee916-4d7c-4735-a3c1-a188b1654deb-openshift-service-ca\") pod \"maas-api-e2e-ait-cd573186-549f84cd45-28nwz\" (UID: \"a90ee916-4d7c-4735-a3c1-a188b1654deb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd573186-549f84cd45-28nwz" Jul 30 00:34:21.388632 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.388481 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a90ee916-4d7c-4735-a3c1-a188b1654deb-maas-api-tls\") pod \"maas-api-e2e-ait-cd573186-549f84cd45-28nwz\" (UID: \"a90ee916-4d7c-4735-a3c1-a188b1654deb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd573186-549f84cd45-28nwz" Jul 30 00:34:21.388632 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.388551 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n59v\" (UniqueName: \"kubernetes.io/projected/a90ee916-4d7c-4735-a3c1-a188b1654deb-kube-api-access-9n59v\") pod \"maas-api-e2e-ait-cd573186-549f84cd45-28nwz\" (UID: \"a90ee916-4d7c-4735-a3c1-a188b1654deb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd573186-549f84cd45-28nwz" Jul 30 00:34:21.422938 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.422907 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:21.489336 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.489307 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a90ee916-4d7c-4735-a3c1-a188b1654deb-maas-api-tls\") pod \"maas-api-e2e-ait-cd573186-549f84cd45-28nwz\" (UID: \"a90ee916-4d7c-4735-a3c1-a188b1654deb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd573186-549f84cd45-28nwz" Jul 30 00:34:21.489490 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.489346 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n59v\" (UniqueName: \"kubernetes.io/projected/a90ee916-4d7c-4735-a3c1-a188b1654deb-kube-api-access-9n59v\") pod \"maas-api-e2e-ait-cd573186-549f84cd45-28nwz\" (UID: \"a90ee916-4d7c-4735-a3c1-a188b1654deb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd573186-549f84cd45-28nwz" Jul 30 00:34:21.489490 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.489400 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a90ee916-4d7c-4735-a3c1-a188b1654deb-openshift-service-ca\") pod \"maas-api-e2e-ait-cd573186-549f84cd45-28nwz\" (UID: \"a90ee916-4d7c-4735-a3c1-a188b1654deb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd573186-549f84cd45-28nwz" Jul 30 00:34:21.490035 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.490016 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a90ee916-4d7c-4735-a3c1-a188b1654deb-openshift-service-ca\") pod \"maas-api-e2e-ait-cd573186-549f84cd45-28nwz\" (UID: \"a90ee916-4d7c-4735-a3c1-a188b1654deb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd573186-549f84cd45-28nwz" Jul 30 00:34:21.491810 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.491787 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a90ee916-4d7c-4735-a3c1-a188b1654deb-maas-api-tls\") pod \"maas-api-e2e-ait-cd573186-549f84cd45-28nwz\" (UID: \"a90ee916-4d7c-4735-a3c1-a188b1654deb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd573186-549f84cd45-28nwz" Jul 30 00:34:21.500455 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.500383 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n59v\" (UniqueName: \"kubernetes.io/projected/a90ee916-4d7c-4735-a3c1-a188b1654deb-kube-api-access-9n59v\") pod \"maas-api-e2e-ait-cd573186-549f84cd45-28nwz\" (UID: \"a90ee916-4d7c-4735-a3c1-a188b1654deb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd573186-549f84cd45-28nwz" Jul 30 00:34:21.565195 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.565162 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd573186-549f84cd45-28nwz" Jul 30 00:34:21.703855 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.703832 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cd573186-549f84cd45-28nwz"] Jul 30 00:34:21.706166 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:34:21.706139 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda90ee916_4d7c_4735_a3c1_a188b1654deb.slice/crio-187e3bca3905d7fc4cbb57db175890de7e998c5ff12a72a6fc30ee05e806b934 WatchSource:0}: Error finding container 187e3bca3905d7fc4cbb57db175890de7e998c5ff12a72a6fc30ee05e806b934: Status 404 returned error can't find the container with id 187e3bca3905d7fc4cbb57db175890de7e998c5ff12a72a6fc30ee05e806b934 Jul 30 00:34:21.707546 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.707461 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 00:34:21.830793 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.830763 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:34:21.858555 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.858504 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:21.952534 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.952504 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:21.994024 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.993994 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hd8v5\" (UniqueName: \"kubernetes.io/projected/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-kube-api-access-hd8v5\") pod \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " Jul 30 00:34:21.994024 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.994027 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istio-token\") pod \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " Jul 30 00:34:21.994265 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.994097 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istio-podinfo\") pod \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " Jul 30 00:34:21.994265 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.994181 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istiod-ca-cert\") pod \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " Jul 30 00:34:21.994265 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.994238 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-workload-certs\") pod \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " Jul 30 00:34:21.994422 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.994269 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istio-data\") pod \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " Jul 30 00:34:21.994422 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.994306 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-workload-socket\") pod \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " Jul 30 00:34:21.994422 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.994345 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-credential-socket\") pod \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " Jul 30 00:34:21.994422 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.994381 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istio-envoy\") pod \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\" (UID: \"e92b1109-04cb-4ad8-87c4-27c6b00c7a13\") " Jul 30 00:34:21.994621 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.994506 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e92b1109-04cb-4ad8-87c4-27c6b00c7a13" (UID: "e92b1109-04cb-4ad8-87c4-27c6b00c7a13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:34:21.994621 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.994611 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-workload-certs\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:34:21.994715 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.994631 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e92b1109-04cb-4ad8-87c4-27c6b00c7a13" (UID: "e92b1109-04cb-4ad8-87c4-27c6b00c7a13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:34:21.994874 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.994851 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e92b1109-04cb-4ad8-87c4-27c6b00c7a13" (UID: "e92b1109-04cb-4ad8-87c4-27c6b00c7a13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:34:21.994978 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.994957 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e92b1109-04cb-4ad8-87c4-27c6b00c7a13" (UID: "e92b1109-04cb-4ad8-87c4-27c6b00c7a13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:34:21.995229 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.995199 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istio-data" (OuterVolumeSpecName: "istio-data") pod "e92b1109-04cb-4ad8-87c4-27c6b00c7a13" (UID: "e92b1109-04cb-4ad8-87c4-27c6b00c7a13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:34:21.996327 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.996305 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e92b1109-04cb-4ad8-87c4-27c6b00c7a13" (UID: "e92b1109-04cb-4ad8-87c4-27c6b00c7a13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 00:34:21.996627 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.996596 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-kube-api-access-hd8v5" (OuterVolumeSpecName: "kube-api-access-hd8v5") pod "e92b1109-04cb-4ad8-87c4-27c6b00c7a13" (UID: "e92b1109-04cb-4ad8-87c4-27c6b00c7a13"). InnerVolumeSpecName "kube-api-access-hd8v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:34:21.996732 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.996677 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e92b1109-04cb-4ad8-87c4-27c6b00c7a13" (UID: "e92b1109-04cb-4ad8-87c4-27c6b00c7a13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:34:21.996732 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:21.996700 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istio-token" (OuterVolumeSpecName: "istio-token") pod "e92b1109-04cb-4ad8-87c4-27c6b00c7a13" (UID: "e92b1109-04cb-4ad8-87c4-27c6b00c7a13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:34:22.095839 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:22.095804 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istio-data\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:34:22.095839 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:22.095842 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-workload-socket\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:34:22.096036 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:22.095859 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-credential-socket\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:34:22.096036 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:22.095874 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istio-envoy\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:34:22.096036 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:22.095889 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd8v5\" (UniqueName: \"kubernetes.io/projected/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-kube-api-access-hd8v5\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:34:22.096036 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:22.095902 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istio-token\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:34:22.096036 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:22.095916 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istio-podinfo\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:34:22.096036 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:22.095928 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92b1109-04cb-4ad8-87c4-27c6b00c7a13-istiod-ca-cert\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:34:22.187076 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:22.187043 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd573186-549f84cd45-28nwz" event={"ID":"a90ee916-4d7c-4735-a3c1-a188b1654deb","Type":"ContainerStarted","Data":"187e3bca3905d7fc4cbb57db175890de7e998c5ff12a72a6fc30ee05e806b934"} Jul 30 00:34:22.188407 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:22.188384 2572 generic.go:356] "Generic (PLEG): container finished" podID="e92b1109-04cb-4ad8-87c4-27c6b00c7a13" containerID="a6a4ed923e0758abce9c6a2dfa86baca47630a3d94f8599d8256527f77ae74e9" exitCode=0 Jul 30 00:34:22.188513 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:22.188440 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" event={"ID":"e92b1109-04cb-4ad8-87c4-27c6b00c7a13","Type":"ContainerDied","Data":"a6a4ed923e0758abce9c6a2dfa86baca47630a3d94f8599d8256527f77ae74e9"} Jul 30 00:34:22.188513 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:22.188458 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" Jul 30 00:34:22.188513 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:22.188478 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2" event={"ID":"e92b1109-04cb-4ad8-87c4-27c6b00c7a13","Type":"ContainerDied","Data":"dd613df708bfa74e6f09c3b3619c58612e6dc1df8725a42e291f154850cfd380"} Jul 30 00:34:22.188513 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:22.188500 2572 scope.go:117] "RemoveContainer" containerID="a6a4ed923e0758abce9c6a2dfa86baca47630a3d94f8599d8256527f77ae74e9" Jul 30 00:34:22.202472 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:22.202450 2572 scope.go:117] "RemoveContainer" containerID="2a59318837be5c6cb7053fb62a956cdbac199a636d8fc8a7055fc719e956ad92" Jul 30 00:34:22.214373 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:22.214348 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2"] Jul 30 00:34:22.220314 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:22.220296 2572 scope.go:117] "RemoveContainer" containerID="a6a4ed923e0758abce9c6a2dfa86baca47630a3d94f8599d8256527f77ae74e9" Jul 30 00:34:22.220611 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:34:22.220586 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6a4ed923e0758abce9c6a2dfa86baca47630a3d94f8599d8256527f77ae74e9\": container with ID starting with a6a4ed923e0758abce9c6a2dfa86baca47630a3d94f8599d8256527f77ae74e9 not found: ID does not exist" containerID="a6a4ed923e0758abce9c6a2dfa86baca47630a3d94f8599d8256527f77ae74e9" Jul 30 00:34:22.220710 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:22.220624 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6a4ed923e0758abce9c6a2dfa86baca47630a3d94f8599d8256527f77ae74e9"} err="failed to get container status \"a6a4ed923e0758abce9c6a2dfa86baca47630a3d94f8599d8256527f77ae74e9\": rpc error: code = NotFound desc = could not find container \"a6a4ed923e0758abce9c6a2dfa86baca47630a3d94f8599d8256527f77ae74e9\": container with ID starting with a6a4ed923e0758abce9c6a2dfa86baca47630a3d94f8599d8256527f77ae74e9 not found: ID does not exist" Jul 30 00:34:22.220710 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:22.220649 2572 scope.go:117] "RemoveContainer" containerID="2a59318837be5c6cb7053fb62a956cdbac199a636d8fc8a7055fc719e956ad92" Jul 30 00:34:22.220932 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:34:22.220910 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a59318837be5c6cb7053fb62a956cdbac199a636d8fc8a7055fc719e956ad92\": container with ID starting with 2a59318837be5c6cb7053fb62a956cdbac199a636d8fc8a7055fc719e956ad92 not found: ID does not exist" containerID="2a59318837be5c6cb7053fb62a956cdbac199a636d8fc8a7055fc719e956ad92" Jul 30 00:34:22.221058 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:22.220941 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a59318837be5c6cb7053fb62a956cdbac199a636d8fc8a7055fc719e956ad92"} err="failed to get container status \"2a59318837be5c6cb7053fb62a956cdbac199a636d8fc8a7055fc719e956ad92\": rpc error: code = NotFound desc = could not find container \"2a59318837be5c6cb7053fb62a956cdbac199a636d8fc8a7055fc719e956ad92\": container with ID starting with 2a59318837be5c6cb7053fb62a956cdbac199a636d8fc8a7055fc719e956ad92 not found: ID does not exist" Jul 30 00:34:22.221289 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:22.221268 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-e5dec626-gw-openshift-default-f9794b548-n5dc2"] Jul 30 00:34:22.992547 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:22.992514 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e92b1109-04cb-4ad8-87c4-27c6b00c7a13" path="/var/lib/kubelet/pods/e92b1109-04cb-4ad8-87c4-27c6b00c7a13/volumes" Jul 30 00:34:23.193884 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:23.193840 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd573186-549f84cd45-28nwz" event={"ID":"a90ee916-4d7c-4735-a3c1-a188b1654deb","Type":"ContainerStarted","Data":"e1eb81a98a59975889a6d2f2a4e2030d7c15943e6ddc1e4034653ee6567eb828"} Jul 30 00:34:23.194084 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:23.193947 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd573186-549f84cd45-28nwz" Jul 30 00:34:23.210956 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:23.210913 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd573186-549f84cd45-28nwz" podStartSLOduration=1.750748728 podStartE2EDuration="2.210898902s" totalImagesPullingTime="460.150174ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:34:23.209463334 +0000 UTC m=+1950.712721594" watchObservedRunningTime="2026-07-30 00:34:23.210898902 +0000 UTC m=+1950.714157161" Jul 30 00:34:29.204451 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:29.204420 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd573186-549f84cd45-28nwz" Jul 30 00:34:51.679626 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:51.679543 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cd573186-549f84cd45-28nwz"] Jul 30 00:34:51.680180 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:51.679784 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd573186-549f84cd45-28nwz" podUID="a90ee916-4d7c-4735-a3c1-a188b1654deb" containerName="maas-api" containerID="cri-o://e1eb81a98a59975889a6d2f2a4e2030d7c15943e6ddc1e4034653ee6567eb828" gracePeriod=30 Jul 30 00:34:51.750911 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:51.750873 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:51.850306 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:51.850272 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:51.933304 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:51.933249 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd573186-549f84cd45-28nwz" Jul 30 00:34:51.951651 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:51.951618 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:52.051075 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:52.051037 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a90ee916-4d7c-4735-a3c1-a188b1654deb-maas-api-tls\") pod \"a90ee916-4d7c-4735-a3c1-a188b1654deb\" (UID: \"a90ee916-4d7c-4735-a3c1-a188b1654deb\") " Jul 30 00:34:52.051268 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:52.051101 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9n59v\" (UniqueName: \"kubernetes.io/projected/a90ee916-4d7c-4735-a3c1-a188b1654deb-kube-api-access-9n59v\") pod \"a90ee916-4d7c-4735-a3c1-a188b1654deb\" (UID: \"a90ee916-4d7c-4735-a3c1-a188b1654deb\") " Jul 30 00:34:52.051268 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:52.051219 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a90ee916-4d7c-4735-a3c1-a188b1654deb-openshift-service-ca\") pod \"a90ee916-4d7c-4735-a3c1-a188b1654deb\" (UID: \"a90ee916-4d7c-4735-a3c1-a188b1654deb\") " Jul 30 00:34:52.051599 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:52.051571 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a90ee916-4d7c-4735-a3c1-a188b1654deb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a90ee916-4d7c-4735-a3c1-a188b1654deb" (UID: "a90ee916-4d7c-4735-a3c1-a188b1654deb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:34:52.053345 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:52.053322 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a90ee916-4d7c-4735-a3c1-a188b1654deb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a90ee916-4d7c-4735-a3c1-a188b1654deb" (UID: "a90ee916-4d7c-4735-a3c1-a188b1654deb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 00:34:52.053442 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:52.053375 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a90ee916-4d7c-4735-a3c1-a188b1654deb-kube-api-access-9n59v" (OuterVolumeSpecName: "kube-api-access-9n59v") pod "a90ee916-4d7c-4735-a3c1-a188b1654deb" (UID: "a90ee916-4d7c-4735-a3c1-a188b1654deb"). InnerVolumeSpecName "kube-api-access-9n59v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:34:52.152598 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:52.152561 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a90ee916-4d7c-4735-a3c1-a188b1654deb-openshift-service-ca\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:34:52.152598 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:52.152591 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a90ee916-4d7c-4735-a3c1-a188b1654deb-maas-api-tls\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:34:52.152789 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:52.152611 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n59v\" (UniqueName: \"kubernetes.io/projected/a90ee916-4d7c-4735-a3c1-a188b1654deb-kube-api-access-9n59v\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:34:52.312248 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:52.312157 2572 generic.go:356] "Generic (PLEG): container finished" podID="a90ee916-4d7c-4735-a3c1-a188b1654deb" containerID="e1eb81a98a59975889a6d2f2a4e2030d7c15943e6ddc1e4034653ee6567eb828" exitCode=0 Jul 30 00:34:52.312248 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:52.312215 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd573186-549f84cd45-28nwz" Jul 30 00:34:52.312248 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:52.312241 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd573186-549f84cd45-28nwz" event={"ID":"a90ee916-4d7c-4735-a3c1-a188b1654deb","Type":"ContainerDied","Data":"e1eb81a98a59975889a6d2f2a4e2030d7c15943e6ddc1e4034653ee6567eb828"} Jul 30 00:34:52.312512 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:52.312293 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cd573186-549f84cd45-28nwz" event={"ID":"a90ee916-4d7c-4735-a3c1-a188b1654deb","Type":"ContainerDied","Data":"187e3bca3905d7fc4cbb57db175890de7e998c5ff12a72a6fc30ee05e806b934"} Jul 30 00:34:52.312512 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:52.312315 2572 scope.go:117] "RemoveContainer" containerID="e1eb81a98a59975889a6d2f2a4e2030d7c15943e6ddc1e4034653ee6567eb828" Jul 30 00:34:52.321220 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:52.321205 2572 scope.go:117] "RemoveContainer" containerID="e1eb81a98a59975889a6d2f2a4e2030d7c15943e6ddc1e4034653ee6567eb828" Jul 30 00:34:52.321459 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:34:52.321441 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1eb81a98a59975889a6d2f2a4e2030d7c15943e6ddc1e4034653ee6567eb828\": container with ID starting with e1eb81a98a59975889a6d2f2a4e2030d7c15943e6ddc1e4034653ee6567eb828 not found: ID does not exist" containerID="e1eb81a98a59975889a6d2f2a4e2030d7c15943e6ddc1e4034653ee6567eb828" Jul 30 00:34:52.321496 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:52.321469 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1eb81a98a59975889a6d2f2a4e2030d7c15943e6ddc1e4034653ee6567eb828"} err="failed to get container status \"e1eb81a98a59975889a6d2f2a4e2030d7c15943e6ddc1e4034653ee6567eb828\": rpc error: code = NotFound desc = could not find container \"e1eb81a98a59975889a6d2f2a4e2030d7c15943e6ddc1e4034653ee6567eb828\": container with ID starting with e1eb81a98a59975889a6d2f2a4e2030d7c15943e6ddc1e4034653ee6567eb828 not found: ID does not exist" Jul 30 00:34:52.333828 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:52.333791 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cd573186-549f84cd45-28nwz"] Jul 30 00:34:52.339298 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:52.339277 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cd573186-549f84cd45-28nwz"] Jul 30 00:34:52.993346 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:52.993314 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a90ee916-4d7c-4735-a3c1-a188b1654deb" path="/var/lib/kubelet/pods/a90ee916-4d7c-4735-a3c1-a188b1654deb/volumes" Jul 30 00:34:53.008072 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.008040 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:53.224977 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.224938 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:53.430185 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.430151 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:53.702310 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.702238 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr"] Jul 30 00:34:53.702554 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.702542 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e92b1109-04cb-4ad8-87c4-27c6b00c7a13" containerName="istio-proxy" Jul 30 00:34:53.702619 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.702556 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e92b1109-04cb-4ad8-87c4-27c6b00c7a13" containerName="istio-proxy" Jul 30 00:34:53.702619 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.702565 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="691163c3-96d1-4a1b-b19a-253e164f943c" containerName="istio-proxy" Jul 30 00:34:53.702619 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.702571 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="691163c3-96d1-4a1b-b19a-253e164f943c" containerName="istio-proxy" Jul 30 00:34:53.702619 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.702579 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e92b1109-04cb-4ad8-87c4-27c6b00c7a13" containerName="istio-proxy" Jul 30 00:34:53.702619 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.702585 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e92b1109-04cb-4ad8-87c4-27c6b00c7a13" containerName="istio-proxy" Jul 30 00:34:53.702619 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.702598 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a90ee916-4d7c-4735-a3c1-a188b1654deb" containerName="maas-api" Jul 30 00:34:53.702619 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.702603 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a90ee916-4d7c-4735-a3c1-a188b1654deb" containerName="maas-api" Jul 30 00:34:53.702821 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.702649 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e92b1109-04cb-4ad8-87c4-27c6b00c7a13" containerName="istio-proxy" Jul 30 00:34:53.702821 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.702657 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e92b1109-04cb-4ad8-87c4-27c6b00c7a13" containerName="istio-proxy" Jul 30 00:34:53.702821 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.702662 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a90ee916-4d7c-4735-a3c1-a188b1654deb" containerName="maas-api" Jul 30 00:34:53.702821 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.702668 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="691163c3-96d1-4a1b-b19a-253e164f943c" containerName="istio-proxy" Jul 30 00:34:53.706792 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.706774 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:53.709533 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.709509 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-2de2fd7b-openshift-default-dockercfg-c5nv4\"" Jul 30 00:34:53.716481 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.716457 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr"] Jul 30 00:34:53.723830 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.723802 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:53.767418 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.767392 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istio-envoy\") pod \"e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:53.767568 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.767422 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istio-data\") pod \"e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:53.767568 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.767440 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istiod-ca-cert\") pod \"e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:53.767568 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.767545 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-workload-socket\") pod \"e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:53.767705 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.767578 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istio-token\") pod \"e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:53.767705 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.767601 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-622qr\" (UniqueName: \"kubernetes.io/projected/54ad616d-42ee-42c3-8c8e-bacd655d78e2-kube-api-access-622qr\") pod \"e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:53.767705 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.767635 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istio-podinfo\") pod \"e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:53.767705 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.767656 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-credential-socket\") pod \"e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:53.767705 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.767682 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-workload-certs\") pod \"e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:53.868794 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.868760 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istio-token\") pod \"e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:53.868794 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.868797 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-622qr\" (UniqueName: \"kubernetes.io/projected/54ad616d-42ee-42c3-8c8e-bacd655d78e2-kube-api-access-622qr\") pod \"e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:53.869036 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.868924 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istio-podinfo\") pod \"e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:53.869036 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.868964 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-credential-socket\") pod \"e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:53.869036 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.868996 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-workload-certs\") pod \"e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:53.869209 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.869052 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istio-envoy\") pod \"e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:53.869209 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.869069 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istio-data\") pod \"e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:53.869209 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.869084 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istiod-ca-cert\") pod \"e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:53.869343 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.869211 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-workload-socket\") pod \"e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:53.869517 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.869488 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-credential-socket\") pod \"e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:53.869636 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.869520 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-workload-certs\") pod \"e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:53.869636 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.869594 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-workload-socket\") pod \"e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:53.869636 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.869609 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istio-data\") pod \"e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:53.869816 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.869800 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istiod-ca-cert\") pod \"e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:53.871833 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.871808 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istio-envoy\") pod \"e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:53.871932 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.871916 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istio-podinfo\") pod \"e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:53.877586 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.877566 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istio-token\") pod \"e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:53.877731 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:53.877714 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-622qr\" (UniqueName: \"kubernetes.io/projected/54ad616d-42ee-42c3-8c8e-bacd655d78e2-kube-api-access-622qr\") pod \"e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:54.016975 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:54.016887 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:54.084077 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:54.084030 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:54.146132 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:54.146088 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr"] Jul 30 00:34:54.148309 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:34:54.148284 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ad616d_42ee_42c3_8c8e_bacd655d78e2.slice/crio-16908e2338a4006aa624509931352fd90718dbde8a61c3ed792472e24bccf2e4 WatchSource:0}: Error finding container 16908e2338a4006aa624509931352fd90718dbde8a61c3ed792472e24bccf2e4: Status 404 returned error can't find the container with id 16908e2338a4006aa624509931352fd90718dbde8a61c3ed792472e24bccf2e4 Jul 30 00:34:54.150390 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:54.150358 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:34:54.150483 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:54.150419 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:34:54.150483 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:54.150451 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:34:54.322169 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:54.322081 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" event={"ID":"54ad616d-42ee-42c3-8c8e-bacd655d78e2","Type":"ContainerStarted","Data":"272b2aa5901e257f78d7e5d006ca275303a2e417fe9d8f0dd1742a2164232baf"} Jul 30 00:34:54.322169 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:54.322158 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" event={"ID":"54ad616d-42ee-42c3-8c8e-bacd655d78e2","Type":"ContainerStarted","Data":"16908e2338a4006aa624509931352fd90718dbde8a61c3ed792472e24bccf2e4"} Jul 30 00:34:54.342238 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:54.342179 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" podStartSLOduration=1.342159035 podStartE2EDuration="1.342159035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:34:54.339817541 +0000 UTC m=+1981.843075801" watchObservedRunningTime="2026-07-30 00:34:54.342159035 +0000 UTC m=+1981.845417296" Jul 30 00:34:55.017174 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:55.017143 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:55.022189 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:55.022163 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:55.325796 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:55.325765 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:55.326715 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:55.326696 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:34:55.721539 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:55.721453 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:55.822465 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:55.822431 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:56.784962 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:56.784929 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7"] Jul 30 00:34:56.787997 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:56.787980 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7" Jul 30 00:34:56.790446 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:56.790425 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-2de2fd7b-serving-cert\"" Jul 30 00:34:56.796588 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:56.796564 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7"] Jul 30 00:34:56.886980 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:56.886951 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:56.895225 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:56.895200 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c1840b4-edc2-41d0-b9a8-0d589b32b851-maas-api-tls\") pod \"maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7\" (UID: \"0c1840b4-edc2-41d0-b9a8-0d589b32b851\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7" Jul 30 00:34:56.895308 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:56.895243 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-775cl\" (UniqueName: \"kubernetes.io/projected/0c1840b4-edc2-41d0-b9a8-0d589b32b851-kube-api-access-775cl\") pod \"maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7\" (UID: \"0c1840b4-edc2-41d0-b9a8-0d589b32b851\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7" Jul 30 00:34:56.895383 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:56.895366 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c1840b4-edc2-41d0-b9a8-0d589b32b851-openshift-service-ca\") pod \"maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7\" (UID: \"0c1840b4-edc2-41d0-b9a8-0d589b32b851\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7" Jul 30 00:34:56.996695 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:56.996662 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c1840b4-edc2-41d0-b9a8-0d589b32b851-openshift-service-ca\") pod \"maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7\" (UID: \"0c1840b4-edc2-41d0-b9a8-0d589b32b851\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7" Jul 30 00:34:56.996874 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:56.996715 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c1840b4-edc2-41d0-b9a8-0d589b32b851-maas-api-tls\") pod \"maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7\" (UID: \"0c1840b4-edc2-41d0-b9a8-0d589b32b851\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7" Jul 30 00:34:56.996874 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:56.996763 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-775cl\" (UniqueName: \"kubernetes.io/projected/0c1840b4-edc2-41d0-b9a8-0d589b32b851-kube-api-access-775cl\") pod \"maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7\" (UID: \"0c1840b4-edc2-41d0-b9a8-0d589b32b851\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7" Jul 30 00:34:56.996995 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:34:56.996897 2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-serving-cert: secret "maas-api-e2e-derive-2de2fd7b-serving-cert" not found Jul 30 00:34:56.996995 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:34:56.996972 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c1840b4-edc2-41d0-b9a8-0d589b32b851-maas-api-tls podName:0c1840b4-edc2-41d0-b9a8-0d589b32b851 nodeName:}" failed. No retries permitted until 2026-07-30 00:34:57.496949194 +0000 UTC m=+1985.000207441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0c1840b4-edc2-41d0-b9a8-0d589b32b851-maas-api-tls") pod "maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7" (UID: "0c1840b4-edc2-41d0-b9a8-0d589b32b851") : secret "maas-api-e2e-derive-2de2fd7b-serving-cert" not found Jul 30 00:34:56.997394 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:56.997374 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c1840b4-edc2-41d0-b9a8-0d589b32b851-openshift-service-ca\") pod \"maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7\" (UID: \"0c1840b4-edc2-41d0-b9a8-0d589b32b851\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7" Jul 30 00:34:57.005298 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:57.005280 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-775cl\" (UniqueName: \"kubernetes.io/projected/0c1840b4-edc2-41d0-b9a8-0d589b32b851-kube-api-access-775cl\") pod \"maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7\" (UID: \"0c1840b4-edc2-41d0-b9a8-0d589b32b851\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7" Jul 30 00:34:57.288042 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:57.288007 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:57.385529 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:57.385489 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:57.460644 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:57.460602 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:34:57.500493 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:57.500460 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c1840b4-edc2-41d0-b9a8-0d589b32b851-maas-api-tls\") pod \"maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7\" (UID: \"0c1840b4-edc2-41d0-b9a8-0d589b32b851\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7" Jul 30 00:34:57.502829 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:57.502807 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c1840b4-edc2-41d0-b9a8-0d589b32b851-maas-api-tls\") pod \"maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7\" (UID: \"0c1840b4-edc2-41d0-b9a8-0d589b32b851\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7" Jul 30 00:34:57.699514 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:57.699481 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7" Jul 30 00:34:57.831364 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:57.828175 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7"] Jul 30 00:34:57.832948 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:34:57.832917 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c1840b4_edc2_41d0_b9a8_0d589b32b851.slice/crio-f2bd66f5813748a2e689ad21c08e54cc339b1a911c9dbf65ca5f52a3c7feb8ab WatchSource:0}: Error finding container f2bd66f5813748a2e689ad21c08e54cc339b1a911c9dbf65ca5f52a3c7feb8ab: Status 404 returned error can't find the container with id f2bd66f5813748a2e689ad21c08e54cc339b1a911c9dbf65ca5f52a3c7feb8ab Jul 30 00:34:58.337796 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:58.337763 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7" event={"ID":"0c1840b4-edc2-41d0-b9a8-0d589b32b851","Type":"ContainerStarted","Data":"f2bd66f5813748a2e689ad21c08e54cc339b1a911c9dbf65ca5f52a3c7feb8ab"} Jul 30 00:34:59.343233 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:59.343196 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7" event={"ID":"0c1840b4-edc2-41d0-b9a8-0d589b32b851","Type":"ContainerStarted","Data":"f7a848ada6946f739ebaef2935daa9f49635e69013b46fcb63c368059a6d00c1"} Jul 30 00:34:59.343614 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:59.343332 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7" Jul 30 00:34:59.360264 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:34:59.360081 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7" podStartSLOduration=2.743336515 podStartE2EDuration="3.360062322s" totalImagesPullingTime="616.725807ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:34:59.3595938 +0000 UTC m=+1986.862852062" watchObservedRunningTime="2026-07-30 00:34:59.360062322 +0000 UTC m=+1986.863320583" Jul 30 00:35:05.356293 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:05.356258 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7" Jul 30 00:35:09.387066 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:09.387025 2572 generic.go:356] "Generic (PLEG): container finished" podID="54ad616d-42ee-42c3-8c8e-bacd655d78e2" containerID="272b2aa5901e257f78d7e5d006ca275303a2e417fe9d8f0dd1742a2164232baf" exitCode=0 Jul 30 00:35:09.387565 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:09.387128 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" event={"ID":"54ad616d-42ee-42c3-8c8e-bacd655d78e2","Type":"ContainerDied","Data":"272b2aa5901e257f78d7e5d006ca275303a2e417fe9d8f0dd1742a2164232baf"} Jul 30 00:35:09.387655 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:09.387635 2572 scope.go:117] "RemoveContainer" containerID="272b2aa5901e257f78d7e5d006ca275303a2e417fe9d8f0dd1742a2164232baf" Jul 30 00:35:09.388852 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:09.388778 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:35:09.388960 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:09.388860 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:35:09.388960 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:09.388906 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:35:10.326343 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:10.326306 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:35:10.393220 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:10.393189 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" event={"ID":"54ad616d-42ee-42c3-8c8e-bacd655d78e2","Type":"ContainerStarted","Data":"372b796651c718954f68f4684b748d4fff7cf1cbfa6cc0a2565e1e22b736abb9"} Jul 30 00:35:11.017810 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:11.017771 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:35:11.018897 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:11.018873 2572 patch_prober.go:28] interesting pod/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Jul 30 00:35:11.018960 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:11.018932 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" podUID="54ad616d-42ee-42c3-8c8e-bacd655d78e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Jul 30 00:35:12.018160 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:12.018095 2572 patch_prober.go:28] interesting pod/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Jul 30 00:35:12.018544 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:12.018179 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" podUID="54ad616d-42ee-42c3-8c8e-bacd655d78e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Jul 30 00:35:13.017819 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:13.017787 2572 patch_prober.go:28] interesting pod/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Jul 30 00:35:13.017994 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:13.017839 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" podUID="54ad616d-42ee-42c3-8c8e-bacd655d78e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Jul 30 00:35:14.017704 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:14.017661 2572 patch_prober.go:28] interesting pod/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Jul 30 00:35:14.018196 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:14.017723 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" podUID="54ad616d-42ee-42c3-8c8e-bacd655d78e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Jul 30 00:35:15.021708 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:15.021681 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:35:15.022082 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:15.021925 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:35:15.022522 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:15.022506 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:35:27.230846 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:27.230805 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7"] Jul 30 00:35:27.231535 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:27.231082 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7" podUID="0c1840b4-edc2-41d0-b9a8-0d589b32b851" containerName="maas-api" containerID="cri-o://f7a848ada6946f739ebaef2935daa9f49635e69013b46fcb63c368059a6d00c1" gracePeriod=30 Jul 30 00:35:27.267000 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:27.266975 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:35:27.367341 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:27.367310 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:35:27.426602 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:27.426572 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:35:27.472547 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:27.472516 2572 generic.go:356] "Generic (PLEG): container finished" podID="0c1840b4-edc2-41d0-b9a8-0d589b32b851" containerID="f7a848ada6946f739ebaef2935daa9f49635e69013b46fcb63c368059a6d00c1" exitCode=0 Jul 30 00:35:27.472677 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:27.472548 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7" event={"ID":"0c1840b4-edc2-41d0-b9a8-0d589b32b851","Type":"ContainerDied","Data":"f7a848ada6946f739ebaef2935daa9f49635e69013b46fcb63c368059a6d00c1"} Jul 30 00:35:27.489357 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:27.489289 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7" Jul 30 00:35:27.553693 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:27.553659 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr"] Jul 30 00:35:27.553905 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:27.553883 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" podUID="54ad616d-42ee-42c3-8c8e-bacd655d78e2" containerName="istio-proxy" containerID="cri-o://372b796651c718954f68f4684b748d4fff7cf1cbfa6cc0a2565e1e22b736abb9" gracePeriod=30 Jul 30 00:35:27.568513 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:27.568489 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-775cl\" (UniqueName: \"kubernetes.io/projected/0c1840b4-edc2-41d0-b9a8-0d589b32b851-kube-api-access-775cl\") pod \"0c1840b4-edc2-41d0-b9a8-0d589b32b851\" (UID: \"0c1840b4-edc2-41d0-b9a8-0d589b32b851\") " Jul 30 00:35:27.568672 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:27.568545 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c1840b4-edc2-41d0-b9a8-0d589b32b851-openshift-service-ca\") pod \"0c1840b4-edc2-41d0-b9a8-0d589b32b851\" (UID: \"0c1840b4-edc2-41d0-b9a8-0d589b32b851\") " Jul 30 00:35:27.568672 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:27.568599 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c1840b4-edc2-41d0-b9a8-0d589b32b851-maas-api-tls\") pod \"0c1840b4-edc2-41d0-b9a8-0d589b32b851\" (UID: \"0c1840b4-edc2-41d0-b9a8-0d589b32b851\") " Jul 30 00:35:27.568931 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:27.568907 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c1840b4-edc2-41d0-b9a8-0d589b32b851-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0c1840b4-edc2-41d0-b9a8-0d589b32b851" (UID: "0c1840b4-edc2-41d0-b9a8-0d589b32b851"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:35:27.570563 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:27.570542 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c1840b4-edc2-41d0-b9a8-0d589b32b851-kube-api-access-775cl" (OuterVolumeSpecName: "kube-api-access-775cl") pod "0c1840b4-edc2-41d0-b9a8-0d589b32b851" (UID: "0c1840b4-edc2-41d0-b9a8-0d589b32b851"). InnerVolumeSpecName "kube-api-access-775cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:35:27.570711 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:27.570695 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c1840b4-edc2-41d0-b9a8-0d589b32b851-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0c1840b4-edc2-41d0-b9a8-0d589b32b851" (UID: "0c1840b4-edc2-41d0-b9a8-0d589b32b851"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 00:35:27.577980 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:27.577956 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:35:27.669394 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:27.669354 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c1840b4-edc2-41d0-b9a8-0d589b32b851-maas-api-tls\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:35:27.669394 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:27.669386 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-775cl\" (UniqueName: \"kubernetes.io/projected/0c1840b4-edc2-41d0-b9a8-0d589b32b851-kube-api-access-775cl\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:35:27.669624 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:27.669403 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c1840b4-edc2-41d0-b9a8-0d589b32b851-openshift-service-ca\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:35:27.799864 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:27.799771 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:35:28.257853 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:28.257821 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:35:28.416262 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:28.416222 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:35:28.478082 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:28.478047 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7" event={"ID":"0c1840b4-edc2-41d0-b9a8-0d589b32b851","Type":"ContainerDied","Data":"f2bd66f5813748a2e689ad21c08e54cc339b1a911c9dbf65ca5f52a3c7feb8ab"} Jul 30 00:35:28.478082 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:28.478064 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7" Jul 30 00:35:28.478349 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:28.478102 2572 scope.go:117] "RemoveContainer" containerID="f7a848ada6946f739ebaef2935daa9f49635e69013b46fcb63c368059a6d00c1" Jul 30 00:35:28.499801 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:28.499777 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7"] Jul 30 00:35:28.503297 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:28.503242 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:35:28.505001 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:28.504979 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2de2fd7b-57d4cd46ff-kz7l7"] Jul 30 00:35:28.992770 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:28.992733 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c1840b4-edc2-41d0-b9a8-0d589b32b851" path="/var/lib/kubelet/pods/0c1840b4-edc2-41d0-b9a8-0d589b32b851/volumes" Jul 30 00:35:29.552326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:29.552288 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:35:29.922795 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:29.922762 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:35:30.024428 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:30.024394 2572 patch_prober.go:28] interesting pod/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 30 00:35:30.024552 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:30.024462 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" podUID="54ad616d-42ee-42c3-8c8e-bacd655d78e2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 30 00:35:32.803446 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:32.803424 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:35:32.918866 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:32.918836 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-workload-certs\") pod \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " Jul 30 00:35:32.919090 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:32.918883 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-622qr\" (UniqueName: \"kubernetes.io/projected/54ad616d-42ee-42c3-8c8e-bacd655d78e2-kube-api-access-622qr\") pod \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " Jul 30 00:35:32.919090 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:32.918913 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-credential-socket\") pod \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " Jul 30 00:35:32.919090 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:32.918937 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-workload-socket\") pod \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " Jul 30 00:35:32.919090 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:32.918963 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istio-token\") pod \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " Jul 30 00:35:32.919090 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:32.919005 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istio-data\") pod \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " Jul 30 00:35:32.919090 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:32.919034 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istiod-ca-cert\") pod \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " Jul 30 00:35:32.919090 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:32.919066 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istio-podinfo\") pod \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " Jul 30 00:35:32.919711 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:32.919121 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istio-envoy\") pod \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\" (UID: \"54ad616d-42ee-42c3-8c8e-bacd655d78e2\") " Jul 30 00:35:32.919711 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:32.919154 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "54ad616d-42ee-42c3-8c8e-bacd655d78e2" (UID: "54ad616d-42ee-42c3-8c8e-bacd655d78e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:35:32.919711 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:32.919271 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "54ad616d-42ee-42c3-8c8e-bacd655d78e2" (UID: "54ad616d-42ee-42c3-8c8e-bacd655d78e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:35:32.919711 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:32.919340 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "54ad616d-42ee-42c3-8c8e-bacd655d78e2" (UID: "54ad616d-42ee-42c3-8c8e-bacd655d78e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:35:32.919711 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:32.919482 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-workload-certs\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:35:32.919711 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:32.919505 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-credential-socket\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:35:32.919711 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:32.919505 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "54ad616d-42ee-42c3-8c8e-bacd655d78e2" (UID: "54ad616d-42ee-42c3-8c8e-bacd655d78e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:35:32.919711 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:32.919523 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-workload-socket\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:35:32.920436 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:32.920404 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istio-data" (OuterVolumeSpecName: "istio-data") pod "54ad616d-42ee-42c3-8c8e-bacd655d78e2" (UID: "54ad616d-42ee-42c3-8c8e-bacd655d78e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:35:32.921404 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:32.921377 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ad616d-42ee-42c3-8c8e-bacd655d78e2-kube-api-access-622qr" (OuterVolumeSpecName: "kube-api-access-622qr") pod "54ad616d-42ee-42c3-8c8e-bacd655d78e2" (UID: "54ad616d-42ee-42c3-8c8e-bacd655d78e2"). InnerVolumeSpecName "kube-api-access-622qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:35:32.921501 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:32.921483 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "54ad616d-42ee-42c3-8c8e-bacd655d78e2" (UID: "54ad616d-42ee-42c3-8c8e-bacd655d78e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:35:32.921718 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:32.921693 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istio-token" (OuterVolumeSpecName: "istio-token") pod "54ad616d-42ee-42c3-8c8e-bacd655d78e2" (UID: "54ad616d-42ee-42c3-8c8e-bacd655d78e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:35:32.921778 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:32.921717 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "54ad616d-42ee-42c3-8c8e-bacd655d78e2" (UID: "54ad616d-42ee-42c3-8c8e-bacd655d78e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 00:35:33.020669 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:33.020637 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istio-data\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:35:33.020669 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:33.020665 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istiod-ca-cert\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:35:33.020669 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:33.020674 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istio-podinfo\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:35:33.020880 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:33.020683 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istio-envoy\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:35:33.020880 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:33.020692 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-622qr\" (UniqueName: \"kubernetes.io/projected/54ad616d-42ee-42c3-8c8e-bacd655d78e2-kube-api-access-622qr\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:35:33.020880 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:33.020704 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ad616d-42ee-42c3-8c8e-bacd655d78e2-istio-token\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:35:33.500797 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:33.500763 2572 generic.go:356] "Generic (PLEG): container finished" podID="54ad616d-42ee-42c3-8c8e-bacd655d78e2" containerID="372b796651c718954f68f4684b748d4fff7cf1cbfa6cc0a2565e1e22b736abb9" exitCode=0 Jul 30 00:35:33.500982 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:33.500812 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" Jul 30 00:35:33.500982 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:33.500848 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" event={"ID":"54ad616d-42ee-42c3-8c8e-bacd655d78e2","Type":"ContainerDied","Data":"372b796651c718954f68f4684b748d4fff7cf1cbfa6cc0a2565e1e22b736abb9"} Jul 30 00:35:33.500982 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:33.500901 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr" event={"ID":"54ad616d-42ee-42c3-8c8e-bacd655d78e2","Type":"ContainerDied","Data":"16908e2338a4006aa624509931352fd90718dbde8a61c3ed792472e24bccf2e4"} Jul 30 00:35:33.500982 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:33.500925 2572 scope.go:117] "RemoveContainer" containerID="372b796651c718954f68f4684b748d4fff7cf1cbfa6cc0a2565e1e22b736abb9" Jul 30 00:35:33.509238 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:33.509220 2572 scope.go:117] "RemoveContainer" containerID="272b2aa5901e257f78d7e5d006ca275303a2e417fe9d8f0dd1742a2164232baf" Jul 30 00:35:33.517255 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:33.517214 2572 scope.go:117] "RemoveContainer" containerID="372b796651c718954f68f4684b748d4fff7cf1cbfa6cc0a2565e1e22b736abb9" Jul 30 00:35:33.517553 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:35:33.517533 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"372b796651c718954f68f4684b748d4fff7cf1cbfa6cc0a2565e1e22b736abb9\": container with ID starting with 372b796651c718954f68f4684b748d4fff7cf1cbfa6cc0a2565e1e22b736abb9 not found: ID does not exist" containerID="372b796651c718954f68f4684b748d4fff7cf1cbfa6cc0a2565e1e22b736abb9" Jul 30 00:35:33.517621 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:33.517561 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"372b796651c718954f68f4684b748d4fff7cf1cbfa6cc0a2565e1e22b736abb9"} err="failed to get container status \"372b796651c718954f68f4684b748d4fff7cf1cbfa6cc0a2565e1e22b736abb9\": rpc error: code = NotFound desc = could not find container \"372b796651c718954f68f4684b748d4fff7cf1cbfa6cc0a2565e1e22b736abb9\": container with ID starting with 372b796651c718954f68f4684b748d4fff7cf1cbfa6cc0a2565e1e22b736abb9 not found: ID does not exist" Jul 30 00:35:33.517621 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:33.517580 2572 scope.go:117] "RemoveContainer" containerID="272b2aa5901e257f78d7e5d006ca275303a2e417fe9d8f0dd1742a2164232baf" Jul 30 00:35:33.517840 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:35:33.517816 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"272b2aa5901e257f78d7e5d006ca275303a2e417fe9d8f0dd1742a2164232baf\": container with ID starting with 272b2aa5901e257f78d7e5d006ca275303a2e417fe9d8f0dd1742a2164232baf not found: ID does not exist" containerID="272b2aa5901e257f78d7e5d006ca275303a2e417fe9d8f0dd1742a2164232baf" Jul 30 00:35:33.517963 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:33.517841 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"272b2aa5901e257f78d7e5d006ca275303a2e417fe9d8f0dd1742a2164232baf"} err="failed to get container status \"272b2aa5901e257f78d7e5d006ca275303a2e417fe9d8f0dd1742a2164232baf\": rpc error: code = NotFound desc = could not find container \"272b2aa5901e257f78d7e5d006ca275303a2e417fe9d8f0dd1742a2164232baf\": container with ID starting with 272b2aa5901e257f78d7e5d006ca275303a2e417fe9d8f0dd1742a2164232baf not found: ID does not exist" Jul 30 00:35:33.518210 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:33.518192 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr"] Jul 30 00:35:33.521522 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:33.521501 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-2de2fd7b-openshift-default-6cdbb766b8-zskcr"] Jul 30 00:35:34.992316 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:34.992278 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54ad616d-42ee-42c3-8c8e-bacd655d78e2" path="/var/lib/kubelet/pods/54ad616d-42ee-42c3-8c8e-bacd655d78e2/volumes" Jul 30 00:35:35.574174 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:35.574137 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:35:35.677005 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:35.676970 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:35:35.860510 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:35.860474 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:35:35.964123 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:35.964091 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:35:36.347595 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:35:36.347566 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:36:02.217563 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:36:02.217524 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:36:08.550126 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:36:08.550068 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:36:08.654693 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:36:08.654663 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:36:08.755863 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:36:08.755829 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:36:24.625083 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:36:24.625044 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:36:30.499019 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:36:30.498988 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:36:30.557165 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:36:30.557129 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:36:30.642158 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:36:30.642102 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:36:30.799014 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:36:30.798908 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:36:41.386150 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:36:41.386086 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:36:41.980919 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:36:41.980886 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:36:47.178471 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:36:47.178427 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:36:48.088754 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:36:48.088720 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:36:48.198010 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:36:48.197974 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:36:48.290074 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:36:48.290033 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:36:48.389295 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:36:48.389198 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:36:53.472517 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:36:53.472484 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:36:53.571093 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:36:53.571040 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:36:54.203236 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:36:54.203203 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:36:55.357579 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:36:55.357513 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:37:01.927119 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:01.927086 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:37:02.027081 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:02.027044 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:37:02.608361 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:02.608333 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:37:08.511080 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:08.511044 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:37:08.612916 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:08.612882 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:37:08.746987 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:08.746953 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:37:09.031261 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.031230 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s"] Jul 30 00:37:09.031628 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.031609 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ad616d-42ee-42c3-8c8e-bacd655d78e2" containerName="istio-proxy" Jul 30 00:37:09.031728 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.031629 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ad616d-42ee-42c3-8c8e-bacd655d78e2" containerName="istio-proxy" Jul 30 00:37:09.031728 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.031648 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c1840b4-edc2-41d0-b9a8-0d589b32b851" containerName="maas-api" Jul 30 00:37:09.031728 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.031656 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c1840b4-edc2-41d0-b9a8-0d589b32b851" containerName="maas-api" Jul 30 00:37:09.031874 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.031773 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="54ad616d-42ee-42c3-8c8e-bacd655d78e2" containerName="istio-proxy" Jul 30 00:37:09.031874 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.031787 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="54ad616d-42ee-42c3-8c8e-bacd655d78e2" containerName="istio-proxy" Jul 30 00:37:09.031874 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.031799 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c1840b4-edc2-41d0-b9a8-0d589b32b851" containerName="maas-api" Jul 30 00:37:09.034660 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.034639 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.037337 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.037321 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8c9d47c1-openshift-default-dockercfg-fl4wh\"" Jul 30 00:37:09.057710 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.057680 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:37:09.059270 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.059248 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s"] Jul 30 00:37:09.126701 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.126668 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21028b78-b70a-4fcc-b002-d4ff2054c305-istio-podinfo\") pod \"e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.126904 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.126713 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-credential-socket\") pod \"e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.126904 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.126763 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-workload-socket\") pod \"e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.126904 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.126807 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-istio-data\") pod \"e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.126904 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.126851 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pvmt\" (UniqueName: \"kubernetes.io/projected/21028b78-b70a-4fcc-b002-d4ff2054c305-kube-api-access-2pvmt\") pod \"e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.127139 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.126930 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-workload-certs\") pod \"e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.127139 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.126951 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21028b78-b70a-4fcc-b002-d4ff2054c305-istio-token\") pod \"e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.127139 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.127012 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-istio-envoy\") pod \"e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.127139 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.127060 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21028b78-b70a-4fcc-b002-d4ff2054c305-istiod-ca-cert\") pod \"e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.227702 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.227666 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pvmt\" (UniqueName: \"kubernetes.io/projected/21028b78-b70a-4fcc-b002-d4ff2054c305-kube-api-access-2pvmt\") pod \"e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.227880 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.227714 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-workload-certs\") pod \"e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.227944 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.227882 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21028b78-b70a-4fcc-b002-d4ff2054c305-istio-token\") pod \"e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.228008 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.227942 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-istio-envoy\") pod \"e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.228067 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.228004 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21028b78-b70a-4fcc-b002-d4ff2054c305-istiod-ca-cert\") pod \"e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.228067 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.228043 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21028b78-b70a-4fcc-b002-d4ff2054c305-istio-podinfo\") pod \"e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.228204 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.228070 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-credential-socket\") pod \"e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.228265 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.228209 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-workload-socket\") pod \"e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.228265 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.228236 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-workload-certs\") pod \"e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.228360 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.228276 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-istio-data\") pod \"e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.228415 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.228386 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-credential-socket\") pod \"e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.228561 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.228530 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-workload-socket\") pod \"e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.228669 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.228640 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-istio-data\") pod \"e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.228736 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.228720 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21028b78-b70a-4fcc-b002-d4ff2054c305-istiod-ca-cert\") pod \"e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.230231 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.230211 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-istio-envoy\") pod \"e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.230498 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.230482 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21028b78-b70a-4fcc-b002-d4ff2054c305-istio-podinfo\") pod \"e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.239296 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.239272 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pvmt\" (UniqueName: \"kubernetes.io/projected/21028b78-b70a-4fcc-b002-d4ff2054c305-kube-api-access-2pvmt\") pod \"e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.241623 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.241603 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21028b78-b70a-4fcc-b002-d4ff2054c305-istio-token\") pod \"e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.343812 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.343777 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:09.458766 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.458735 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:37:09.481992 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.481968 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s"] Jul 30 00:37:09.483516 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:37:09.483479 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21028b78_b70a_4fcc_b002_d4ff2054c305.slice/crio-80df205c7616bc16602d7130e9e134c362309049ecc6f70e10547d6c373ad116 WatchSource:0}: Error finding container 80df205c7616bc16602d7130e9e134c362309049ecc6f70e10547d6c373ad116: Status 404 returned error can't find the container with id 80df205c7616bc16602d7130e9e134c362309049ecc6f70e10547d6c373ad116 Jul 30 00:37:09.485599 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.485570 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:37:09.485670 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.485637 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:37:09.485670 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.485662 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:37:09.881655 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.881622 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" event={"ID":"21028b78-b70a-4fcc-b002-d4ff2054c305","Type":"ContainerStarted","Data":"7fc418225970fd2979724e8c0eba08bde26be1118925de4633036e718d9244ec"} Jul 30 00:37:09.882030 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.881676 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" event={"ID":"21028b78-b70a-4fcc-b002-d4ff2054c305","Type":"ContainerStarted","Data":"80df205c7616bc16602d7130e9e134c362309049ecc6f70e10547d6c373ad116"} Jul 30 00:37:09.902687 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:09.902638 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" podStartSLOduration=0.902621951 podStartE2EDuration="902.621951ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:37:09.901222581 +0000 UTC m=+2117.404480844" watchObservedRunningTime="2026-07-30 00:37:09.902621951 +0000 UTC m=+2117.405880211" Jul 30 00:37:10.344503 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:10.344469 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:10.349462 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:10.349439 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:10.885827 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:10.885779 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:10.886920 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:10.886894 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:11.793942 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:11.793907 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:37:11.894724 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:11.894681 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:37:12.884689 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:12.884651 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6"] Jul 30 00:37:12.885139 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:12.885098 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ad616d-42ee-42c3-8c8e-bacd655d78e2" containerName="istio-proxy" Jul 30 00:37:12.885139 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:12.885139 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ad616d-42ee-42c3-8c8e-bacd655d78e2" containerName="istio-proxy" Jul 30 00:37:12.888348 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:12.888327 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6" Jul 30 00:37:12.890783 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:12.890763 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8c9d47c1-serving-cert\"" Jul 30 00:37:12.898855 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:12.898830 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6"] Jul 30 00:37:12.916311 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:12.916286 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:37:12.962654 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:12.962624 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f1c84dc-fa0d-4465-ac99-c4a26f36697f-openshift-service-ca\") pod \"maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6\" (UID: \"8f1c84dc-fa0d-4465-ac99-c4a26f36697f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6" Jul 30 00:37:12.962796 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:12.962690 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgb7b\" (UniqueName: \"kubernetes.io/projected/8f1c84dc-fa0d-4465-ac99-c4a26f36697f-kube-api-access-hgb7b\") pod \"maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6\" (UID: \"8f1c84dc-fa0d-4465-ac99-c4a26f36697f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6" Jul 30 00:37:12.962860 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:12.962814 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f1c84dc-fa0d-4465-ac99-c4a26f36697f-maas-api-tls\") pod \"maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6\" (UID: \"8f1c84dc-fa0d-4465-ac99-c4a26f36697f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6" Jul 30 00:37:13.063994 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:13.063961 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgb7b\" (UniqueName: \"kubernetes.io/projected/8f1c84dc-fa0d-4465-ac99-c4a26f36697f-kube-api-access-hgb7b\") pod \"maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6\" (UID: \"8f1c84dc-fa0d-4465-ac99-c4a26f36697f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6" Jul 30 00:37:13.064189 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:13.064009 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f1c84dc-fa0d-4465-ac99-c4a26f36697f-maas-api-tls\") pod \"maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6\" (UID: \"8f1c84dc-fa0d-4465-ac99-c4a26f36697f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6" Jul 30 00:37:13.064189 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:13.064057 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f1c84dc-fa0d-4465-ac99-c4a26f36697f-openshift-service-ca\") pod \"maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6\" (UID: \"8f1c84dc-fa0d-4465-ac99-c4a26f36697f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6" Jul 30 00:37:13.064309 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:37:13.064288 2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-serving-cert: secret "maas-api-e2e-mt-8c9d47c1-serving-cert" not found Jul 30 00:37:13.064395 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:37:13.064384 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f1c84dc-fa0d-4465-ac99-c4a26f36697f-maas-api-tls podName:8f1c84dc-fa0d-4465-ac99-c4a26f36697f nodeName:}" failed. No retries permitted until 2026-07-30 00:37:13.564363057 +0000 UTC m=+2121.067621305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8f1c84dc-fa0d-4465-ac99-c4a26f36697f-maas-api-tls") pod "maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6" (UID: "8f1c84dc-fa0d-4465-ac99-c4a26f36697f") : secret "maas-api-e2e-mt-8c9d47c1-serving-cert" not found Jul 30 00:37:13.064660 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:13.064643 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f1c84dc-fa0d-4465-ac99-c4a26f36697f-openshift-service-ca\") pod \"maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6\" (UID: \"8f1c84dc-fa0d-4465-ac99-c4a26f36697f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6" Jul 30 00:37:13.080239 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:13.080219 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgb7b\" (UniqueName: \"kubernetes.io/projected/8f1c84dc-fa0d-4465-ac99-c4a26f36697f-kube-api-access-hgb7b\") pod \"maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6\" (UID: \"8f1c84dc-fa0d-4465-ac99-c4a26f36697f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6" Jul 30 00:37:13.280694 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:13.280607 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:37:13.501569 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:13.501533 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:37:13.566826 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:13.566728 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:37:13.568741 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:13.568714 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f1c84dc-fa0d-4465-ac99-c4a26f36697f-maas-api-tls\") pod \"maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6\" (UID: \"8f1c84dc-fa0d-4465-ac99-c4a26f36697f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6" Jul 30 00:37:13.571126 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:13.571093 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f1c84dc-fa0d-4465-ac99-c4a26f36697f-maas-api-tls\") pod \"maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6\" (UID: \"8f1c84dc-fa0d-4465-ac99-c4a26f36697f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6" Jul 30 00:37:13.801022 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:13.800984 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6" Jul 30 00:37:13.931655 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:13.931633 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6"] Jul 30 00:37:13.933888 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:37:13.933857 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f1c84dc_fa0d_4465_ac99_c4a26f36697f.slice/crio-6d73ff8914e968f8d404f0c7eb0b378f2cf8cfbbef63cb36e365a63c37a067c1 WatchSource:0}: Error finding container 6d73ff8914e968f8d404f0c7eb0b378f2cf8cfbbef63cb36e365a63c37a067c1: Status 404 returned error can't find the container with id 6d73ff8914e968f8d404f0c7eb0b378f2cf8cfbbef63cb36e365a63c37a067c1 Jul 30 00:37:14.907503 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:14.907461 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6" event={"ID":"8f1c84dc-fa0d-4465-ac99-c4a26f36697f","Type":"ContainerStarted","Data":"b9d1f97ec20fe83957191c3c600a259aa0196b319bf0f2e53389618f82743def"} Jul 30 00:37:14.907655 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:14.907563 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6" Jul 30 00:37:14.907655 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:14.907582 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6" event={"ID":"8f1c84dc-fa0d-4465-ac99-c4a26f36697f","Type":"ContainerStarted","Data":"6d73ff8914e968f8d404f0c7eb0b378f2cf8cfbbef63cb36e365a63c37a067c1"} Jul 30 00:37:14.923915 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:14.923862 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6" podStartSLOduration=2.401276927 podStartE2EDuration="2.923848197s" totalImagesPullingTime="522.57127ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:37:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:37:14.922187281 +0000 UTC m=+2122.425445544" watchObservedRunningTime="2026-07-30 00:37:14.923848197 +0000 UTC m=+2122.427106464" Jul 30 00:37:20.916814 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:20.916789 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6" Jul 30 00:37:26.464321 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:26.464283 2572 patch_prober.go:28] interesting pod/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": read tcp 10.133.0.2:36360->10.133.0.28:15021: read: connection reset by peer" start-of-body= Jul 30 00:37:26.464725 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:26.464380 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" podUID="21028b78-b70a-4fcc-b002-d4ff2054c305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": read tcp 10.133.0.2:36360->10.133.0.28:15021: read: connection reset by peer" Jul 30 00:37:26.963353 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:26.963314 2572 generic.go:356] "Generic (PLEG): container finished" podID="21028b78-b70a-4fcc-b002-d4ff2054c305" containerID="7fc418225970fd2979724e8c0eba08bde26be1118925de4633036e718d9244ec" exitCode=0 Jul 30 00:37:26.963531 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:26.963387 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" event={"ID":"21028b78-b70a-4fcc-b002-d4ff2054c305","Type":"ContainerDied","Data":"7fc418225970fd2979724e8c0eba08bde26be1118925de4633036e718d9244ec"} Jul 30 00:37:26.963794 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:26.963781 2572 scope.go:117] "RemoveContainer" containerID="7fc418225970fd2979724e8c0eba08bde26be1118925de4633036e718d9244ec" Jul 30 00:37:26.964769 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:26.964742 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:37:26.964832 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:26.964809 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:37:26.964871 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:26.964840 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:37:27.969234 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:27.969194 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" event={"ID":"21028b78-b70a-4fcc-b002-d4ff2054c305","Type":"ContainerStarted","Data":"5bb85d731778478c1b997c861f39cbbb45bd826e6815fd6e50b852d4c75f4929"} Jul 30 00:37:28.344439 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:28.344395 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:28.344761 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:28.344468 2572 patch_prober.go:28] interesting pod/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Jul 30 00:37:28.344949 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:28.344918 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" podUID="21028b78-b70a-4fcc-b002-d4ff2054c305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Jul 30 00:37:29.345235 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:29.345190 2572 patch_prober.go:28] interesting pod/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Jul 30 00:37:29.345753 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:29.345268 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" podUID="21028b78-b70a-4fcc-b002-d4ff2054c305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Jul 30 00:37:30.344508 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:30.344467 2572 patch_prober.go:28] interesting pod/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Jul 30 00:37:30.344727 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:30.344551 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" podUID="21028b78-b70a-4fcc-b002-d4ff2054c305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Jul 30 00:37:31.345411 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:31.344933 2572 patch_prober.go:28] interesting pod/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Jul 30 00:37:31.345411 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:31.345002 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" podUID="21028b78-b70a-4fcc-b002-d4ff2054c305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Jul 30 00:37:32.347663 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:32.347632 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:32.348028 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:32.348010 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:32.348766 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:32.348747 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:37:36.787408 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:36.787371 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:37:37.421179 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:37.421145 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:37:37.521975 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:37.521937 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:37:37.860930 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:37.860898 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:37:43.549624 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:43.549587 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:37:45.244716 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:45.244608 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:37:54.330954 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:54.330922 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6"] Jul 30 00:37:54.331641 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:54.331152 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6" podUID="8f1c84dc-fa0d-4465-ac99-c4a26f36697f" containerName="maas-api" containerID="cri-o://b9d1f97ec20fe83957191c3c600a259aa0196b319bf0f2e53389618f82743def" gracePeriod=30 Jul 30 00:37:54.350924 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:54.350890 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:37:54.427535 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:54.427504 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:37:54.489181 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:54.489135 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:37:54.543780 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:54.543733 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:37:54.584764 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:54.584705 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6" Jul 30 00:37:54.627796 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:54.627766 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgb7b\" (UniqueName: \"kubernetes.io/projected/8f1c84dc-fa0d-4465-ac99-c4a26f36697f-kube-api-access-hgb7b\") pod \"8f1c84dc-fa0d-4465-ac99-c4a26f36697f\" (UID: \"8f1c84dc-fa0d-4465-ac99-c4a26f36697f\") " Jul 30 00:37:54.627972 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:54.627826 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f1c84dc-fa0d-4465-ac99-c4a26f36697f-openshift-service-ca\") pod \"8f1c84dc-fa0d-4465-ac99-c4a26f36697f\" (UID: \"8f1c84dc-fa0d-4465-ac99-c4a26f36697f\") " Jul 30 00:37:54.627972 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:54.627847 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f1c84dc-fa0d-4465-ac99-c4a26f36697f-maas-api-tls\") pod \"8f1c84dc-fa0d-4465-ac99-c4a26f36697f\" (UID: \"8f1c84dc-fa0d-4465-ac99-c4a26f36697f\") " Jul 30 00:37:54.628208 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:54.628176 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f1c84dc-fa0d-4465-ac99-c4a26f36697f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8f1c84dc-fa0d-4465-ac99-c4a26f36697f" (UID: "8f1c84dc-fa0d-4465-ac99-c4a26f36697f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:37:54.629823 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:54.629799 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f1c84dc-fa0d-4465-ac99-c4a26f36697f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8f1c84dc-fa0d-4465-ac99-c4a26f36697f" (UID: "8f1c84dc-fa0d-4465-ac99-c4a26f36697f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 00:37:54.629928 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:54.629905 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f1c84dc-fa0d-4465-ac99-c4a26f36697f-kube-api-access-hgb7b" (OuterVolumeSpecName: "kube-api-access-hgb7b") pod "8f1c84dc-fa0d-4465-ac99-c4a26f36697f" (UID: "8f1c84dc-fa0d-4465-ac99-c4a26f36697f"). InnerVolumeSpecName "kube-api-access-hgb7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:37:54.728519 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:54.728485 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f1c84dc-fa0d-4465-ac99-c4a26f36697f-openshift-service-ca\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:37:54.728519 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:54.728514 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f1c84dc-fa0d-4465-ac99-c4a26f36697f-maas-api-tls\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:37:54.728519 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:54.728523 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgb7b\" (UniqueName: \"kubernetes.io/projected/8f1c84dc-fa0d-4465-ac99-c4a26f36697f-kube-api-access-hgb7b\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:37:54.897926 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:54.897833 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:37:55.085979 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:55.085944 2572 generic.go:356] "Generic (PLEG): container finished" podID="8f1c84dc-fa0d-4465-ac99-c4a26f36697f" containerID="b9d1f97ec20fe83957191c3c600a259aa0196b319bf0f2e53389618f82743def" exitCode=0 Jul 30 00:37:55.086161 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:55.085992 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6" Jul 30 00:37:55.086161 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:55.086004 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6" event={"ID":"8f1c84dc-fa0d-4465-ac99-c4a26f36697f","Type":"ContainerDied","Data":"b9d1f97ec20fe83957191c3c600a259aa0196b319bf0f2e53389618f82743def"} Jul 30 00:37:55.086161 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:55.086051 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6" event={"ID":"8f1c84dc-fa0d-4465-ac99-c4a26f36697f","Type":"ContainerDied","Data":"6d73ff8914e968f8d404f0c7eb0b378f2cf8cfbbef63cb36e365a63c37a067c1"} Jul 30 00:37:55.086161 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:55.086073 2572 scope.go:117] "RemoveContainer" containerID="b9d1f97ec20fe83957191c3c600a259aa0196b319bf0f2e53389618f82743def" Jul 30 00:37:55.094085 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:55.094059 2572 scope.go:117] "RemoveContainer" containerID="b9d1f97ec20fe83957191c3c600a259aa0196b319bf0f2e53389618f82743def" Jul 30 00:37:55.094348 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:37:55.094331 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9d1f97ec20fe83957191c3c600a259aa0196b319bf0f2e53389618f82743def\": container with ID starting with b9d1f97ec20fe83957191c3c600a259aa0196b319bf0f2e53389618f82743def not found: ID does not exist" containerID="b9d1f97ec20fe83957191c3c600a259aa0196b319bf0f2e53389618f82743def" Jul 30 00:37:55.094412 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:55.094356 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9d1f97ec20fe83957191c3c600a259aa0196b319bf0f2e53389618f82743def"} err="failed to get container status \"b9d1f97ec20fe83957191c3c600a259aa0196b319bf0f2e53389618f82743def\": rpc error: code = NotFound desc = could not find container \"b9d1f97ec20fe83957191c3c600a259aa0196b319bf0f2e53389618f82743def\": container with ID starting with b9d1f97ec20fe83957191c3c600a259aa0196b319bf0f2e53389618f82743def not found: ID does not exist" Jul 30 00:37:55.106254 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:55.106231 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6"] Jul 30 00:37:55.110621 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:55.110600 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8c9d47c1-5779f6578-5rcb6"] Jul 30 00:37:56.992476 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:37:56.992444 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f1c84dc-fa0d-4465-ac99-c4a26f36697f" path="/var/lib/kubelet/pods/8f1c84dc-fa0d-4465-ac99-c4a26f36697f/volumes" Jul 30 00:38:00.554150 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:00.554092 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:38:00.657165 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:00.657134 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:38:00.788360 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:00.788327 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s"] Jul 30 00:38:00.788613 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:00.788589 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" podUID="21028b78-b70a-4fcc-b002-d4ff2054c305" containerName="istio-proxy" containerID="cri-o://5bb85d731778478c1b997c861f39cbbb45bd826e6815fd6e50b852d4c75f4929" gracePeriod=30 Jul 30 00:38:00.807828 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:00.807762 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:38:01.038502 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:01.038448 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:38:02.319703 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:02.319664 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:38:02.350348 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:02.350314 2572 patch_prober.go:28] interesting pod/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 30 00:38:02.350505 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:02.350386 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" podUID="21028b78-b70a-4fcc-b002-d4ff2054c305" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 30 00:38:02.420355 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:02.420324 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:38:06.051505 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.051482 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:38:06.130077 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.129776 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pvmt\" (UniqueName: \"kubernetes.io/projected/21028b78-b70a-4fcc-b002-d4ff2054c305-kube-api-access-2pvmt\") pod \"21028b78-b70a-4fcc-b002-d4ff2054c305\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " Jul 30 00:38:06.130077 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.129835 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-workload-certs\") pod \"21028b78-b70a-4fcc-b002-d4ff2054c305\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " Jul 30 00:38:06.130077 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.129871 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-istio-envoy\") pod \"21028b78-b70a-4fcc-b002-d4ff2054c305\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " Jul 30 00:38:06.130077 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.129958 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-credential-socket\") pod \"21028b78-b70a-4fcc-b002-d4ff2054c305\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " Jul 30 00:38:06.130077 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.129983 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21028b78-b70a-4fcc-b002-d4ff2054c305-istiod-ca-cert\") pod \"21028b78-b70a-4fcc-b002-d4ff2054c305\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " Jul 30 00:38:06.130077 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.130014 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21028b78-b70a-4fcc-b002-d4ff2054c305-istio-token\") pod \"21028b78-b70a-4fcc-b002-d4ff2054c305\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " Jul 30 00:38:06.130077 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.130047 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-workload-socket\") pod \"21028b78-b70a-4fcc-b002-d4ff2054c305\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " Jul 30 00:38:06.130614 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.130095 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-istio-data\") pod \"21028b78-b70a-4fcc-b002-d4ff2054c305\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " Jul 30 00:38:06.130614 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.130168 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21028b78-b70a-4fcc-b002-d4ff2054c305-istio-podinfo\") pod \"21028b78-b70a-4fcc-b002-d4ff2054c305\" (UID: \"21028b78-b70a-4fcc-b002-d4ff2054c305\") " Jul 30 00:38:06.131424 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.131383 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21028b78-b70a-4fcc-b002-d4ff2054c305-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "21028b78-b70a-4fcc-b002-d4ff2054c305" (UID: "21028b78-b70a-4fcc-b002-d4ff2054c305"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:38:06.132029 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.131995 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "21028b78-b70a-4fcc-b002-d4ff2054c305" (UID: "21028b78-b70a-4fcc-b002-d4ff2054c305"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:38:06.132282 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.132253 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "21028b78-b70a-4fcc-b002-d4ff2054c305" (UID: "21028b78-b70a-4fcc-b002-d4ff2054c305"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:38:06.132524 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.132499 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "21028b78-b70a-4fcc-b002-d4ff2054c305" (UID: "21028b78-b70a-4fcc-b002-d4ff2054c305"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:38:06.133023 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.132995 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-istio-data" (OuterVolumeSpecName: "istio-data") pod "21028b78-b70a-4fcc-b002-d4ff2054c305" (UID: "21028b78-b70a-4fcc-b002-d4ff2054c305"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:38:06.136963 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.136794 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21028b78-b70a-4fcc-b002-d4ff2054c305-kube-api-access-2pvmt" (OuterVolumeSpecName: "kube-api-access-2pvmt") pod "21028b78-b70a-4fcc-b002-d4ff2054c305" (UID: "21028b78-b70a-4fcc-b002-d4ff2054c305"). InnerVolumeSpecName "kube-api-access-2pvmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:38:06.137131 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.137083 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "21028b78-b70a-4fcc-b002-d4ff2054c305" (UID: "21028b78-b70a-4fcc-b002-d4ff2054c305"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:38:06.137210 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.137169 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21028b78-b70a-4fcc-b002-d4ff2054c305-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "21028b78-b70a-4fcc-b002-d4ff2054c305" (UID: "21028b78-b70a-4fcc-b002-d4ff2054c305"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 00:38:06.137920 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.137895 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21028b78-b70a-4fcc-b002-d4ff2054c305-istio-token" (OuterVolumeSpecName: "istio-token") pod "21028b78-b70a-4fcc-b002-d4ff2054c305" (UID: "21028b78-b70a-4fcc-b002-d4ff2054c305"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:38:06.140478 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.140451 2572 generic.go:356] "Generic (PLEG): container finished" podID="21028b78-b70a-4fcc-b002-d4ff2054c305" containerID="5bb85d731778478c1b997c861f39cbbb45bd826e6815fd6e50b852d4c75f4929" exitCode=0 Jul 30 00:38:06.140631 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.140515 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" Jul 30 00:38:06.140631 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.140540 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" event={"ID":"21028b78-b70a-4fcc-b002-d4ff2054c305","Type":"ContainerDied","Data":"5bb85d731778478c1b997c861f39cbbb45bd826e6815fd6e50b852d4c75f4929"} Jul 30 00:38:06.140744 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.140651 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s" event={"ID":"21028b78-b70a-4fcc-b002-d4ff2054c305","Type":"ContainerDied","Data":"80df205c7616bc16602d7130e9e134c362309049ecc6f70e10547d6c373ad116"} Jul 30 00:38:06.140744 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.140693 2572 scope.go:117] "RemoveContainer" containerID="5bb85d731778478c1b997c861f39cbbb45bd826e6815fd6e50b852d4c75f4929" Jul 30 00:38:06.153897 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.153881 2572 scope.go:117] "RemoveContainer" containerID="7fc418225970fd2979724e8c0eba08bde26be1118925de4633036e718d9244ec" Jul 30 00:38:06.161013 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.160996 2572 scope.go:117] "RemoveContainer" containerID="5bb85d731778478c1b997c861f39cbbb45bd826e6815fd6e50b852d4c75f4929" Jul 30 00:38:06.161283 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:38:06.161252 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bb85d731778478c1b997c861f39cbbb45bd826e6815fd6e50b852d4c75f4929\": container with ID starting with 5bb85d731778478c1b997c861f39cbbb45bd826e6815fd6e50b852d4c75f4929 not found: ID does not exist" containerID="5bb85d731778478c1b997c861f39cbbb45bd826e6815fd6e50b852d4c75f4929" Jul 30 00:38:06.161372 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.161289 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bb85d731778478c1b997c861f39cbbb45bd826e6815fd6e50b852d4c75f4929"} err="failed to get container status \"5bb85d731778478c1b997c861f39cbbb45bd826e6815fd6e50b852d4c75f4929\": rpc error: code = NotFound desc = could not find container \"5bb85d731778478c1b997c861f39cbbb45bd826e6815fd6e50b852d4c75f4929\": container with ID starting with 5bb85d731778478c1b997c861f39cbbb45bd826e6815fd6e50b852d4c75f4929 not found: ID does not exist" Jul 30 00:38:06.161372 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.161308 2572 scope.go:117] "RemoveContainer" containerID="7fc418225970fd2979724e8c0eba08bde26be1118925de4633036e718d9244ec" Jul 30 00:38:06.161639 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:38:06.161616 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fc418225970fd2979724e8c0eba08bde26be1118925de4633036e718d9244ec\": container with ID starting with 7fc418225970fd2979724e8c0eba08bde26be1118925de4633036e718d9244ec not found: ID does not exist" containerID="7fc418225970fd2979724e8c0eba08bde26be1118925de4633036e718d9244ec" Jul 30 00:38:06.161733 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.161643 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fc418225970fd2979724e8c0eba08bde26be1118925de4633036e718d9244ec"} err="failed to get container status \"7fc418225970fd2979724e8c0eba08bde26be1118925de4633036e718d9244ec\": rpc error: code = NotFound desc = could not find container \"7fc418225970fd2979724e8c0eba08bde26be1118925de4633036e718d9244ec\": container with ID starting with 7fc418225970fd2979724e8c0eba08bde26be1118925de4633036e718d9244ec not found: ID does not exist" Jul 30 00:38:06.164030 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.164003 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s"] Jul 30 00:38:06.167636 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.167614 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8c9d47c1-openshift-default-69769c468d-z8b9s"] Jul 30 00:38:06.231126 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.231085 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-credential-socket\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:38:06.231126 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.231127 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21028b78-b70a-4fcc-b002-d4ff2054c305-istiod-ca-cert\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:38:06.231276 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.231139 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21028b78-b70a-4fcc-b002-d4ff2054c305-istio-token\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:38:06.231276 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.231148 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-workload-socket\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:38:06.231276 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.231156 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-istio-data\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:38:06.231276 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.231164 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21028b78-b70a-4fcc-b002-d4ff2054c305-istio-podinfo\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:38:06.231276 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.231173 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pvmt\" (UniqueName: \"kubernetes.io/projected/21028b78-b70a-4fcc-b002-d4ff2054c305-kube-api-access-2pvmt\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:38:06.231276 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.231181 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-workload-certs\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:38:06.231276 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.231190 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21028b78-b70a-4fcc-b002-d4ff2054c305-istio-envoy\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:38:06.881308 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.881260 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:38:06.934756 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.934731 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:38:06.992603 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:06.992575 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21028b78-b70a-4fcc-b002-d4ff2054c305" path="/var/lib/kubelet/pods/21028b78-b70a-4fcc-b002-d4ff2054c305/volumes" Jul 30 00:38:07.009027 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:07.009000 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:38:07.111862 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:07.111827 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:38:07.217034 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:07.216948 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:38:12.979820 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:12.979761 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:38:13.080228 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:13.080196 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:38:19.804329 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:19.804297 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:38:20.070071 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.069990 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq"] Jul 30 00:38:20.070332 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.070319 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21028b78-b70a-4fcc-b002-d4ff2054c305" containerName="istio-proxy" Jul 30 00:38:20.070377 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.070334 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="21028b78-b70a-4fcc-b002-d4ff2054c305" containerName="istio-proxy" Jul 30 00:38:20.070377 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.070346 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f1c84dc-fa0d-4465-ac99-c4a26f36697f" containerName="maas-api" Jul 30 00:38:20.070377 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.070352 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f1c84dc-fa0d-4465-ac99-c4a26f36697f" containerName="maas-api" Jul 30 00:38:20.070377 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.070363 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21028b78-b70a-4fcc-b002-d4ff2054c305" containerName="istio-proxy" Jul 30 00:38:20.070377 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.070371 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="21028b78-b70a-4fcc-b002-d4ff2054c305" containerName="istio-proxy" Jul 30 00:38:20.070563 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.070448 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="21028b78-b70a-4fcc-b002-d4ff2054c305" containerName="istio-proxy" Jul 30 00:38:20.070563 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.070459 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="21028b78-b70a-4fcc-b002-d4ff2054c305" containerName="istio-proxy" Jul 30 00:38:20.070563 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.070469 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f1c84dc-fa0d-4465-ac99-c4a26f36697f" containerName="maas-api" Jul 30 00:38:20.075195 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.075174 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.077893 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.077870 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-99c2ecca-openshift-default-dockercfg-ktxg6\"" Jul 30 00:38:20.084018 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.083997 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq"] Jul 30 00:38:20.096378 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.096348 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:38:20.151663 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.151635 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istio-envoy\") pod \"e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.151663 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.151667 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istio-token\") pod \"e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.151860 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.151712 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istio-podinfo\") pod \"e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.151860 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.151737 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hjlk\" (UniqueName: \"kubernetes.io/projected/b0ba13ec-f05e-41b5-a745-5b7413ea158c-kube-api-access-7hjlk\") pod \"e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.151860 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.151760 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istiod-ca-cert\") pod \"e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.151860 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.151785 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-workload-socket\") pod \"e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.151985 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.151858 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-credential-socket\") pod \"e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.151985 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.151890 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istio-data\") pod \"e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.151985 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.151909 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-workload-certs\") pod \"e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.253277 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.253233 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istio-podinfo\") pod \"e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.253485 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.253294 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hjlk\" (UniqueName: \"kubernetes.io/projected/b0ba13ec-f05e-41b5-a745-5b7413ea158c-kube-api-access-7hjlk\") pod \"e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.253485 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.253326 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istiod-ca-cert\") pod \"e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.253485 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.253357 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-workload-socket\") pod \"e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.253485 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.253398 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-credential-socket\") pod \"e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.253485 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.253427 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istio-data\") pod \"e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.253485 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.253451 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-workload-certs\") pod \"e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.253806 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.253500 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istio-envoy\") pod \"e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.253806 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.253525 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istio-token\") pod \"e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.253953 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.253928 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-workload-certs\") pod \"e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.254032 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.253961 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istio-data\") pod \"e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.254307 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.254264 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-credential-socket\") pod \"e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.254307 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.254294 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istiod-ca-cert\") pod \"e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.254481 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.254308 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-workload-socket\") pod \"e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.256095 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.256069 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istio-envoy\") pod \"e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.256376 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.256359 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istio-podinfo\") pod \"e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.261294 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.261269 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istio-token\") pod \"e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.261649 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.261626 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hjlk\" (UniqueName: \"kubernetes.io/projected/b0ba13ec-f05e-41b5-a745-5b7413ea158c-kube-api-access-7hjlk\") pod \"e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.386086 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.386051 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:20.414940 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.414904 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:38:20.518438 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.518414 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq"] Jul 30 00:38:20.520450 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:38:20.520419 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ba13ec_f05e_41b5_a745_5b7413ea158c.slice/crio-5c59619225b955d54c22e397a4a99605c6f8294ca5d7b0e46b972d3b9663b6f9 WatchSource:0}: Error finding container 5c59619225b955d54c22e397a4a99605c6f8294ca5d7b0e46b972d3b9663b6f9: Status 404 returned error can't find the container with id 5c59619225b955d54c22e397a4a99605c6f8294ca5d7b0e46b972d3b9663b6f9 Jul 30 00:38:20.522792 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.522762 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:38:20.522889 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.522835 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:38:20.522889 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:20.522876 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:38:21.201200 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:21.201162 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" event={"ID":"b0ba13ec-f05e-41b5-a745-5b7413ea158c","Type":"ContainerStarted","Data":"1e7a4458034f5ef5ca125abb3ddc2a8014fab47ab3425cd4b509490bed129b8c"} Jul 30 00:38:21.201579 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:21.201228 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" event={"ID":"b0ba13ec-f05e-41b5-a745-5b7413ea158c","Type":"ContainerStarted","Data":"5c59619225b955d54c22e397a4a99605c6f8294ca5d7b0e46b972d3b9663b6f9"} Jul 30 00:38:21.222413 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:21.222364 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" podStartSLOduration=1.222347969 podStartE2EDuration="1.222347969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:38:21.220860779 +0000 UTC m=+2188.724119041" watchObservedRunningTime="2026-07-30 00:38:21.222347969 +0000 UTC m=+2188.725606229" Jul 30 00:38:21.387043 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:21.387005 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:21.391542 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:21.391520 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:22.204998 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:22.204968 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:22.205946 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:22.205924 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:23.181762 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:23.181719 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:38:23.280192 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:23.280159 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:38:24.268491 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:24.268461 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc"] Jul 30 00:38:24.271742 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:24.271719 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc" Jul 30 00:38:24.274351 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:24.274330 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-99c2ecca-serving-cert\"" Jul 30 00:38:24.279135 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:24.279082 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc"] Jul 30 00:38:24.302137 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:24.302099 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:38:24.389495 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:24.389454 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zclf\" (UniqueName: \"kubernetes.io/projected/73045086-5695-451e-850e-7e73afc5143d-kube-api-access-7zclf\") pod \"maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc\" (UID: \"73045086-5695-451e-850e-7e73afc5143d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc" Jul 30 00:38:24.389495 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:24.389506 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73045086-5695-451e-850e-7e73afc5143d-maas-api-tls\") pod \"maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc\" (UID: \"73045086-5695-451e-850e-7e73afc5143d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc" Jul 30 00:38:24.389719 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:24.389577 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73045086-5695-451e-850e-7e73afc5143d-openshift-service-ca\") pod \"maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc\" (UID: \"73045086-5695-451e-850e-7e73afc5143d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc" Jul 30 00:38:24.490400 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:24.490353 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73045086-5695-451e-850e-7e73afc5143d-openshift-service-ca\") pod \"maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc\" (UID: \"73045086-5695-451e-850e-7e73afc5143d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc" Jul 30 00:38:24.490591 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:24.490451 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zclf\" (UniqueName: \"kubernetes.io/projected/73045086-5695-451e-850e-7e73afc5143d-kube-api-access-7zclf\") pod \"maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc\" (UID: \"73045086-5695-451e-850e-7e73afc5143d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc" Jul 30 00:38:24.490591 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:24.490491 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73045086-5695-451e-850e-7e73afc5143d-maas-api-tls\") pod \"maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc\" (UID: \"73045086-5695-451e-850e-7e73afc5143d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc" Jul 30 00:38:24.490704 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:38:24.490642 2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-serving-cert: secret "maas-api-e2e-mt-99c2ecca-serving-cert" not found Jul 30 00:38:24.490761 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:38:24.490714 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73045086-5695-451e-850e-7e73afc5143d-maas-api-tls podName:73045086-5695-451e-850e-7e73afc5143d nodeName:}" failed. No retries permitted until 2026-07-30 00:38:24.990691747 +0000 UTC m=+2192.493949999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/73045086-5695-451e-850e-7e73afc5143d-maas-api-tls") pod "maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc" (UID: "73045086-5695-451e-850e-7e73afc5143d") : secret "maas-api-e2e-mt-99c2ecca-serving-cert" not found Jul 30 00:38:24.490984 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:24.490962 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73045086-5695-451e-850e-7e73afc5143d-openshift-service-ca\") pod \"maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc\" (UID: \"73045086-5695-451e-850e-7e73afc5143d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc" Jul 30 00:38:24.501489 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:24.501466 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zclf\" (UniqueName: \"kubernetes.io/projected/73045086-5695-451e-850e-7e73afc5143d-kube-api-access-7zclf\") pod \"maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc\" (UID: \"73045086-5695-451e-850e-7e73afc5143d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc" Jul 30 00:38:24.864179 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:24.864143 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:38:24.995082 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:24.995052 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73045086-5695-451e-850e-7e73afc5143d-maas-api-tls\") pod \"maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc\" (UID: \"73045086-5695-451e-850e-7e73afc5143d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc" Jul 30 00:38:24.997416 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:24.997391 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73045086-5695-451e-850e-7e73afc5143d-maas-api-tls\") pod \"maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc\" (UID: \"73045086-5695-451e-850e-7e73afc5143d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc" Jul 30 00:38:25.016475 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:25.016442 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:38:25.112902 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:25.112867 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:38:25.184225 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:25.184137 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc" Jul 30 00:38:25.312856 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:25.312820 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc"] Jul 30 00:38:25.315674 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:38:25.315645 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73045086_5695_451e_850e_7e73afc5143d.slice/crio-deab80171c757a66922b0ce8ab286b3032d2ed51dfba8b2da882bad8f2bba85a WatchSource:0}: Error finding container deab80171c757a66922b0ce8ab286b3032d2ed51dfba8b2da882bad8f2bba85a: Status 404 returned error can't find the container with id deab80171c757a66922b0ce8ab286b3032d2ed51dfba8b2da882bad8f2bba85a Jul 30 00:38:26.225359 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:26.225329 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc" event={"ID":"73045086-5695-451e-850e-7e73afc5143d","Type":"ContainerStarted","Data":"e18885313c6a71b1ef37418b221b1d35b1f38d23a0d51ad56d64ca394c6a93e7"} Jul 30 00:38:26.225509 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:26.225378 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc" event={"ID":"73045086-5695-451e-850e-7e73afc5143d","Type":"ContainerStarted","Data":"deab80171c757a66922b0ce8ab286b3032d2ed51dfba8b2da882bad8f2bba85a"} Jul 30 00:38:26.225509 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:26.225433 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc" Jul 30 00:38:26.242520 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:26.242470 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc" podStartSLOduration=1.810617712 podStartE2EDuration="2.242454334s" totalImagesPullingTime="431.836622ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:38:26.240595677 +0000 UTC m=+2193.743853937" watchObservedRunningTime="2026-07-30 00:38:26.242454334 +0000 UTC m=+2193.745712593" Jul 30 00:38:32.242173 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:32.242141 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc" Jul 30 00:38:36.274799 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:36.274765 2572 generic.go:356] "Generic (PLEG): container finished" podID="b0ba13ec-f05e-41b5-a745-5b7413ea158c" containerID="1e7a4458034f5ef5ca125abb3ddc2a8014fab47ab3425cd4b509490bed129b8c" exitCode=0 Jul 30 00:38:36.275340 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:36.274844 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" event={"ID":"b0ba13ec-f05e-41b5-a745-5b7413ea158c","Type":"ContainerDied","Data":"1e7a4458034f5ef5ca125abb3ddc2a8014fab47ab3425cd4b509490bed129b8c"} Jul 30 00:38:36.275407 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:36.275371 2572 scope.go:117] "RemoveContainer" containerID="1e7a4458034f5ef5ca125abb3ddc2a8014fab47ab3425cd4b509490bed129b8c" Jul 30 00:38:36.276369 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:36.276344 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:38:36.276469 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:36.276411 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:38:36.276469 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:36.276441 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:38:37.205119 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:37.205079 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:37.280695 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:37.280659 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" event={"ID":"b0ba13ec-f05e-41b5-a745-5b7413ea158c","Type":"ContainerStarted","Data":"8d4acc3a1febfcaa4f52c620051847d370ca4dbf523dce3fb9fca327bf1518e6"} Jul 30 00:38:37.387213 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:37.387164 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:37.388389 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:37.388361 2572 patch_prober.go:28] interesting pod/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 30 00:38:37.388514 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:37.388422 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" podUID="b0ba13ec-f05e-41b5-a745-5b7413ea158c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 30 00:38:38.387177 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:38.387135 2572 patch_prober.go:28] interesting pod/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 30 00:38:38.387596 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:38.387191 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" podUID="b0ba13ec-f05e-41b5-a745-5b7413ea158c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 30 00:38:39.387253 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:39.387216 2572 patch_prober.go:28] interesting pod/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 30 00:38:39.387638 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:39.387277 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" podUID="b0ba13ec-f05e-41b5-a745-5b7413ea158c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 30 00:38:40.387007 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:40.386974 2572 patch_prober.go:28] interesting pod/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 30 00:38:40.387204 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:40.387028 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" podUID="b0ba13ec-f05e-41b5-a745-5b7413ea158c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 30 00:38:41.390780 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:41.390749 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:41.391191 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:41.391088 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:41.391706 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:41.391689 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:38:48.081357 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:48.081323 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:38:57.022999 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:38:57.022967 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:39:32.677595 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:39:32.677559 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc"] Jul 30 00:39:32.678169 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:39:32.677814 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc" podUID="73045086-5695-451e-850e-7e73afc5143d" containerName="maas-api" containerID="cri-o://e18885313c6a71b1ef37418b221b1d35b1f38d23a0d51ad56d64ca394c6a93e7" gracePeriod=30 Jul 30 00:39:32.700530 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:39:32.700503 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:39:32.936320 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:39:32.936257 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc" Jul 30 00:39:33.003699 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:39:33.003673 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zclf\" (UniqueName: \"kubernetes.io/projected/73045086-5695-451e-850e-7e73afc5143d-kube-api-access-7zclf\") pod \"73045086-5695-451e-850e-7e73afc5143d\" (UID: \"73045086-5695-451e-850e-7e73afc5143d\") " Jul 30 00:39:33.003699 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:39:33.003702 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73045086-5695-451e-850e-7e73afc5143d-openshift-service-ca\") pod \"73045086-5695-451e-850e-7e73afc5143d\" (UID: \"73045086-5695-451e-850e-7e73afc5143d\") " Jul 30 00:39:33.003890 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:39:33.003729 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73045086-5695-451e-850e-7e73afc5143d-maas-api-tls\") pod \"73045086-5695-451e-850e-7e73afc5143d\" (UID: \"73045086-5695-451e-850e-7e73afc5143d\") " Jul 30 00:39:33.004098 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:39:33.004071 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73045086-5695-451e-850e-7e73afc5143d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "73045086-5695-451e-850e-7e73afc5143d" (UID: "73045086-5695-451e-850e-7e73afc5143d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:39:33.005804 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:39:33.005777 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73045086-5695-451e-850e-7e73afc5143d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "73045086-5695-451e-850e-7e73afc5143d" (UID: "73045086-5695-451e-850e-7e73afc5143d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 00:39:33.005911 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:39:33.005825 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73045086-5695-451e-850e-7e73afc5143d-kube-api-access-7zclf" (OuterVolumeSpecName: "kube-api-access-7zclf") pod "73045086-5695-451e-850e-7e73afc5143d" (UID: "73045086-5695-451e-850e-7e73afc5143d"). InnerVolumeSpecName "kube-api-access-7zclf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:39:33.105090 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:39:33.105056 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zclf\" (UniqueName: \"kubernetes.io/projected/73045086-5695-451e-850e-7e73afc5143d-kube-api-access-7zclf\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:39:33.105090 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:39:33.105082 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73045086-5695-451e-850e-7e73afc5143d-openshift-service-ca\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:39:33.105090 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:39:33.105093 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73045086-5695-451e-850e-7e73afc5143d-maas-api-tls\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:39:33.289622 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:39:33.289529 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:39:33.390487 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:39:33.390451 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:39:33.512928 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:39:33.512890 2572 generic.go:356] "Generic (PLEG): container finished" podID="73045086-5695-451e-850e-7e73afc5143d" containerID="e18885313c6a71b1ef37418b221b1d35b1f38d23a0d51ad56d64ca394c6a93e7" exitCode=0 Jul 30 00:39:33.513092 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:39:33.512939 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc" Jul 30 00:39:33.513092 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:39:33.512982 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc" event={"ID":"73045086-5695-451e-850e-7e73afc5143d","Type":"ContainerDied","Data":"e18885313c6a71b1ef37418b221b1d35b1f38d23a0d51ad56d64ca394c6a93e7"} Jul 30 00:39:33.513092 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:39:33.513037 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc" event={"ID":"73045086-5695-451e-850e-7e73afc5143d","Type":"ContainerDied","Data":"deab80171c757a66922b0ce8ab286b3032d2ed51dfba8b2da882bad8f2bba85a"} Jul 30 00:39:33.513092 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:39:33.513069 2572 scope.go:117] "RemoveContainer" containerID="e18885313c6a71b1ef37418b221b1d35b1f38d23a0d51ad56d64ca394c6a93e7" Jul 30 00:39:33.521519 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:39:33.521503 2572 scope.go:117] "RemoveContainer" containerID="e18885313c6a71b1ef37418b221b1d35b1f38d23a0d51ad56d64ca394c6a93e7" Jul 30 00:39:33.521773 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:39:33.521754 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e18885313c6a71b1ef37418b221b1d35b1f38d23a0d51ad56d64ca394c6a93e7\": container with ID starting with e18885313c6a71b1ef37418b221b1d35b1f38d23a0d51ad56d64ca394c6a93e7 not found: ID does not exist" containerID="e18885313c6a71b1ef37418b221b1d35b1f38d23a0d51ad56d64ca394c6a93e7" Jul 30 00:39:33.521843 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:39:33.521781 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e18885313c6a71b1ef37418b221b1d35b1f38d23a0d51ad56d64ca394c6a93e7"} err="failed to get container status \"e18885313c6a71b1ef37418b221b1d35b1f38d23a0d51ad56d64ca394c6a93e7\": rpc error: code = NotFound desc = could not find container \"e18885313c6a71b1ef37418b221b1d35b1f38d23a0d51ad56d64ca394c6a93e7\": container with ID starting with e18885313c6a71b1ef37418b221b1d35b1f38d23a0d51ad56d64ca394c6a93e7 not found: ID does not exist" Jul 30 00:39:33.535698 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:39:33.535668 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc"] Jul 30 00:39:33.539548 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:39:33.539526 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-99c2ecca-56447c6984-nz4gc"] Jul 30 00:39:34.992420 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:39:34.992390 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73045086-5695-451e-850e-7e73afc5143d" path="/var/lib/kubelet/pods/73045086-5695-451e-850e-7e73afc5143d/volumes" Jul 30 00:39:36.040715 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:39:36.040672 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:39:36.709259 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:39:36.709220 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:39:41.068450 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:39:41.068410 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:40:17.487478 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:17.487421 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:40:18.174090 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:18.174057 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:40:23.585796 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:23.585751 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:40:23.685003 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:23.684965 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:40:23.835404 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:23.835373 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq"] Jul 30 00:40:23.835611 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:23.835581 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" podUID="b0ba13ec-f05e-41b5-a745-5b7413ea158c" containerName="istio-proxy" containerID="cri-o://8d4acc3a1febfcaa4f52c620051847d370ca4dbf523dce3fb9fca327bf1518e6" gracePeriod=30 Jul 30 00:40:24.047884 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:24.047851 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:40:24.149095 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:24.149011 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:40:25.145305 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:25.145272 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:40:25.455073 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:25.454997 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:40:25.560549 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:25.560512 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:40:26.001495 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:26.001460 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:40:26.114511 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:26.114477 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:40:26.241677 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:26.241639 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:40:26.393344 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:26.393308 2572 patch_prober.go:28] interesting pod/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 30 00:40:26.393517 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:26.393379 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" podUID="b0ba13ec-f05e-41b5-a745-5b7413ea158c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 30 00:40:27.574198 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:27.574163 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:40:29.086430 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.086407 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:40:29.169445 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.169370 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istio-podinfo\") pod \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " Jul 30 00:40:29.169445 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.169407 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-workload-socket\") pod \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " Jul 30 00:40:29.169445 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.169434 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hjlk\" (UniqueName: \"kubernetes.io/projected/b0ba13ec-f05e-41b5-a745-5b7413ea158c-kube-api-access-7hjlk\") pod \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " Jul 30 00:40:29.169710 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.169499 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-workload-certs\") pod \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " Jul 30 00:40:29.169710 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.169532 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istiod-ca-cert\") pod \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " Jul 30 00:40:29.169710 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.169568 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istio-token\") pod \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " Jul 30 00:40:29.169710 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.169609 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istio-envoy\") pod \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " Jul 30 00:40:29.169710 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.169636 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-credential-socket\") pod \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " Jul 30 00:40:29.169710 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.169679 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istio-data\") pod \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\" (UID: \"b0ba13ec-f05e-41b5-a745-5b7413ea158c\") " Jul 30 00:40:29.169710 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.169679 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b0ba13ec-f05e-41b5-a745-5b7413ea158c" (UID: "b0ba13ec-f05e-41b5-a745-5b7413ea158c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:40:29.170077 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.169899 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b0ba13ec-f05e-41b5-a745-5b7413ea158c" (UID: "b0ba13ec-f05e-41b5-a745-5b7413ea158c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:40:29.170077 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.169971 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-workload-socket\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:40:29.170218 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.170196 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b0ba13ec-f05e-41b5-a745-5b7413ea158c" (UID: "b0ba13ec-f05e-41b5-a745-5b7413ea158c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:40:29.170277 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.170225 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b0ba13ec-f05e-41b5-a745-5b7413ea158c" (UID: "b0ba13ec-f05e-41b5-a745-5b7413ea158c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:40:29.171084 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.171057 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istio-data" (OuterVolumeSpecName: "istio-data") pod "b0ba13ec-f05e-41b5-a745-5b7413ea158c" (UID: "b0ba13ec-f05e-41b5-a745-5b7413ea158c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:40:29.171846 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.171820 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ba13ec-f05e-41b5-a745-5b7413ea158c-kube-api-access-7hjlk" (OuterVolumeSpecName: "kube-api-access-7hjlk") pod "b0ba13ec-f05e-41b5-a745-5b7413ea158c" (UID: "b0ba13ec-f05e-41b5-a745-5b7413ea158c"). InnerVolumeSpecName "kube-api-access-7hjlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:40:29.171846 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.171834 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b0ba13ec-f05e-41b5-a745-5b7413ea158c" (UID: "b0ba13ec-f05e-41b5-a745-5b7413ea158c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 00:40:29.172073 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.172057 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istio-token" (OuterVolumeSpecName: "istio-token") pod "b0ba13ec-f05e-41b5-a745-5b7413ea158c" (UID: "b0ba13ec-f05e-41b5-a745-5b7413ea158c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:40:29.172207 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.172126 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b0ba13ec-f05e-41b5-a745-5b7413ea158c" (UID: "b0ba13ec-f05e-41b5-a745-5b7413ea158c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:40:29.270870 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.270832 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istio-data\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:40:29.270870 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.270865 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istio-podinfo\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:40:29.270870 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.270878 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hjlk\" (UniqueName: \"kubernetes.io/projected/b0ba13ec-f05e-41b5-a745-5b7413ea158c-kube-api-access-7hjlk\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:40:29.271149 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.270890 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-workload-certs\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:40:29.271149 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.270902 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istiod-ca-cert\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:40:29.271149 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.270912 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istio-token\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:40:29.271149 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.270922 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-istio-envoy\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:40:29.271149 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.270932 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ba13ec-f05e-41b5-a745-5b7413ea158c-credential-socket\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:40:29.461451 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.461354 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:40:29.743328 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.743238 2572 generic.go:356] "Generic (PLEG): container finished" podID="b0ba13ec-f05e-41b5-a745-5b7413ea158c" containerID="8d4acc3a1febfcaa4f52c620051847d370ca4dbf523dce3fb9fca327bf1518e6" exitCode=0 Jul 30 00:40:29.743328 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.743291 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" Jul 30 00:40:29.743528 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.743292 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" event={"ID":"b0ba13ec-f05e-41b5-a745-5b7413ea158c","Type":"ContainerDied","Data":"8d4acc3a1febfcaa4f52c620051847d370ca4dbf523dce3fb9fca327bf1518e6"} Jul 30 00:40:29.743528 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.743416 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq" event={"ID":"b0ba13ec-f05e-41b5-a745-5b7413ea158c","Type":"ContainerDied","Data":"5c59619225b955d54c22e397a4a99605c6f8294ca5d7b0e46b972d3b9663b6f9"} Jul 30 00:40:29.743528 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.743444 2572 scope.go:117] "RemoveContainer" containerID="8d4acc3a1febfcaa4f52c620051847d370ca4dbf523dce3fb9fca327bf1518e6" Jul 30 00:40:29.752397 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.752383 2572 scope.go:117] "RemoveContainer" containerID="1e7a4458034f5ef5ca125abb3ddc2a8014fab47ab3425cd4b509490bed129b8c" Jul 30 00:40:29.759533 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.759515 2572 scope.go:117] "RemoveContainer" containerID="8d4acc3a1febfcaa4f52c620051847d370ca4dbf523dce3fb9fca327bf1518e6" Jul 30 00:40:29.759786 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:40:29.759767 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d4acc3a1febfcaa4f52c620051847d370ca4dbf523dce3fb9fca327bf1518e6\": container with ID starting with 8d4acc3a1febfcaa4f52c620051847d370ca4dbf523dce3fb9fca327bf1518e6 not found: ID does not exist" containerID="8d4acc3a1febfcaa4f52c620051847d370ca4dbf523dce3fb9fca327bf1518e6" Jul 30 00:40:29.759837 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.759794 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d4acc3a1febfcaa4f52c620051847d370ca4dbf523dce3fb9fca327bf1518e6"} err="failed to get container status \"8d4acc3a1febfcaa4f52c620051847d370ca4dbf523dce3fb9fca327bf1518e6\": rpc error: code = NotFound desc = could not find container \"8d4acc3a1febfcaa4f52c620051847d370ca4dbf523dce3fb9fca327bf1518e6\": container with ID starting with 8d4acc3a1febfcaa4f52c620051847d370ca4dbf523dce3fb9fca327bf1518e6 not found: ID does not exist" Jul 30 00:40:29.759837 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.759811 2572 scope.go:117] "RemoveContainer" containerID="1e7a4458034f5ef5ca125abb3ddc2a8014fab47ab3425cd4b509490bed129b8c" Jul 30 00:40:29.760013 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:40:29.759999 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e7a4458034f5ef5ca125abb3ddc2a8014fab47ab3425cd4b509490bed129b8c\": container with ID starting with 1e7a4458034f5ef5ca125abb3ddc2a8014fab47ab3425cd4b509490bed129b8c not found: ID does not exist" containerID="1e7a4458034f5ef5ca125abb3ddc2a8014fab47ab3425cd4b509490bed129b8c" Jul 30 00:40:29.760056 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.760017 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e7a4458034f5ef5ca125abb3ddc2a8014fab47ab3425cd4b509490bed129b8c"} err="failed to get container status \"1e7a4458034f5ef5ca125abb3ddc2a8014fab47ab3425cd4b509490bed129b8c\": rpc error: code = NotFound desc = could not find container \"1e7a4458034f5ef5ca125abb3ddc2a8014fab47ab3425cd4b509490bed129b8c\": container with ID starting with 1e7a4458034f5ef5ca125abb3ddc2a8014fab47ab3425cd4b509490bed129b8c not found: ID does not exist" Jul 30 00:40:29.793158 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.793130 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq"] Jul 30 00:40:29.803062 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:29.803035 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-99c2ecca-openshift-default-56c5947d44-jwpqq"] Jul 30 00:40:30.991569 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:30.991540 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0ba13ec-f05e-41b5-a745-5b7413ea158c" path="/var/lib/kubelet/pods/b0ba13ec-f05e-41b5-a745-5b7413ea158c/volumes" Jul 30 00:40:33.348855 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:33.348818 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:40:33.406683 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:33.406647 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:40:33.996637 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:33.996599 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:40:40.064369 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:40.064334 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:40:40.168484 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:40.168454 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:40:40.264318 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:40.264278 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:40:50.768131 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:50.768027 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:40:51.089649 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:40:51.089617 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:41:02.103267 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:02.103225 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:41:02.150475 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:02.150442 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:41:12.024071 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:12.024038 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:41:18.235132 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.235074 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:41:18.335907 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.335861 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:41:18.583693 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.583645 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:41:18.847281 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.847210 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd"] Jul 30 00:41:18.847540 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.847528 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73045086-5695-451e-850e-7e73afc5143d" containerName="maas-api" Jul 30 00:41:18.847582 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.847542 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="73045086-5695-451e-850e-7e73afc5143d" containerName="maas-api" Jul 30 00:41:18.847582 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.847561 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0ba13ec-f05e-41b5-a745-5b7413ea158c" containerName="istio-proxy" Jul 30 00:41:18.847582 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.847566 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0ba13ec-f05e-41b5-a745-5b7413ea158c" containerName="istio-proxy" Jul 30 00:41:18.847582 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.847579 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0ba13ec-f05e-41b5-a745-5b7413ea158c" containerName="istio-proxy" Jul 30 00:41:18.847710 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.847587 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0ba13ec-f05e-41b5-a745-5b7413ea158c" containerName="istio-proxy" Jul 30 00:41:18.847710 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.847634 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0ba13ec-f05e-41b5-a745-5b7413ea158c" containerName="istio-proxy" Jul 30 00:41:18.847710 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.847642 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="73045086-5695-451e-850e-7e73afc5143d" containerName="maas-api" Jul 30 00:41:18.850420 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.850404 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:18.852676 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.852658 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d11ca7-openshift-default-dockercfg-kn85d\"" Jul 30 00:41:18.862415 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.862391 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd"] Jul 30 00:41:18.869688 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.869664 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:41:18.877804 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.877784 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istiod-ca-cert\") pod \"e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:18.877971 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.877832 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-workload-socket\") pod \"e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:18.877971 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.877861 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istio-envoy\") pod \"e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:18.877971 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.877901 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istio-data\") pod \"e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:18.877971 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.877929 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrnnz\" (UniqueName: \"kubernetes.io/projected/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-kube-api-access-nrnnz\") pod \"e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:18.878294 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.877997 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istio-podinfo\") pod \"e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:18.878294 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.878045 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-credential-socket\") pod \"e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:18.878294 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.878073 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istio-token\") pod \"e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:18.878294 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.878146 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-workload-certs\") pod \"e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:18.978712 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.978678 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-workload-socket\") pod \"e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:18.978897 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.978722 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istio-envoy\") pod \"e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:18.978897 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.978740 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istio-data\") pod \"e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:18.978897 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.978758 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrnnz\" (UniqueName: \"kubernetes.io/projected/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-kube-api-access-nrnnz\") pod \"e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:18.978897 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.978790 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istio-podinfo\") pod \"e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:18.978897 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.978822 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-credential-socket\") pod \"e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:18.978897 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.978845 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istio-token\") pod \"e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:18.978897 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.978891 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-workload-certs\") pod \"e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:18.979279 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.978939 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istiod-ca-cert\") pod \"e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:18.979339 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.979312 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-workload-socket\") pod \"e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:18.979390 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.979339 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istio-data\") pod \"e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:18.979484 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.979450 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-workload-certs\") pod \"e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:18.979639 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.979609 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-credential-socket\") pod \"e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:18.979693 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.979664 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istiod-ca-cert\") pod \"e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:18.981129 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.981093 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istio-envoy\") pod \"e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:18.981859 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.981836 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istio-podinfo\") pod \"e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:18.986607 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.986582 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istio-token\") pod \"e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:18.987277 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:18.987253 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrnnz\" (UniqueName: \"kubernetes.io/projected/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-kube-api-access-nrnnz\") pod \"e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:19.004207 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:19.004181 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:41:19.161085 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:19.161003 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:19.289963 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:19.289938 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd"] Jul 30 00:41:19.291950 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:41:19.291914 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7589aa8b_bf6b_47e7_ac1b_926de232c2e4.slice/crio-1813e09342e792e7fea07deab625b9bde0a0fb9d5441e6ef0043d9685f58c6ae WatchSource:0}: Error finding container 1813e09342e792e7fea07deab625b9bde0a0fb9d5441e6ef0043d9685f58c6ae: Status 404 returned error can't find the container with id 1813e09342e792e7fea07deab625b9bde0a0fb9d5441e6ef0043d9685f58c6ae Jul 30 00:41:19.294061 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:19.294026 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:41:19.294145 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:19.294123 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:41:19.294195 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:19.294165 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:41:19.940049 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:19.940016 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" event={"ID":"7589aa8b-bf6b-47e7-ac1b-926de232c2e4","Type":"ContainerStarted","Data":"d5059f746492a75cda8ed5d7a021afce46f5df1ed810651bdc2cb946002eb0b3"} Jul 30 00:41:19.940222 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:19.940067 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" event={"ID":"7589aa8b-bf6b-47e7-ac1b-926de232c2e4","Type":"ContainerStarted","Data":"1813e09342e792e7fea07deab625b9bde0a0fb9d5441e6ef0043d9685f58c6ae"} Jul 30 00:41:19.965923 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:19.965871 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" podStartSLOduration=1.965854721 podStartE2EDuration="1.965854721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:41:19.963680324 +0000 UTC m=+2367.466938594" watchObservedRunningTime="2026-07-30 00:41:19.965854721 +0000 UTC m=+2367.469112981" Jul 30 00:41:20.161851 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:20.161824 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:20.166430 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:20.166407 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:20.944340 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:20.944315 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:20.945209 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:20.945189 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:21.884029 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:21.883980 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:41:22.355010 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:22.354979 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:41:23.404489 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:23.404450 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:41:23.589530 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:23.589493 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:41:23.687686 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:23.687603 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:41:30.056916 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:30.056885 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:41:30.155705 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:30.155673 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:41:35.002326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:35.002236 2572 generic.go:356] "Generic (PLEG): container finished" podID="7589aa8b-bf6b-47e7-ac1b-926de232c2e4" containerID="d5059f746492a75cda8ed5d7a021afce46f5df1ed810651bdc2cb946002eb0b3" exitCode=0 Jul 30 00:41:35.002326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:35.002286 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" event={"ID":"7589aa8b-bf6b-47e7-ac1b-926de232c2e4","Type":"ContainerDied","Data":"d5059f746492a75cda8ed5d7a021afce46f5df1ed810651bdc2cb946002eb0b3"} Jul 30 00:41:35.002797 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:35.002640 2572 scope.go:117] "RemoveContainer" containerID="d5059f746492a75cda8ed5d7a021afce46f5df1ed810651bdc2cb946002eb0b3" Jul 30 00:41:35.003627 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:35.003598 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:41:35.003700 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:35.003683 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:41:35.003743 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:35.003730 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:41:35.944730 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:35.944699 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:36.010471 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:36.010442 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" event={"ID":"7589aa8b-bf6b-47e7-ac1b-926de232c2e4","Type":"ContainerStarted","Data":"3c9e24b9ce2737e0057acc1ae7b2850f7fea1dbedda6c9dd637491ac16cf0cb0"} Jul 30 00:41:36.161909 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:36.161879 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:36.162932 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:36.162911 2572 patch_prober.go:28] interesting pod/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Jul 30 00:41:36.162990 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:36.162957 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" podUID="7589aa8b-bf6b-47e7-ac1b-926de232c2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Jul 30 00:41:37.162436 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:37.162400 2572 patch_prober.go:28] interesting pod/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Jul 30 00:41:37.162828 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:37.162456 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" podUID="7589aa8b-bf6b-47e7-ac1b-926de232c2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Jul 30 00:41:38.162354 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:38.162316 2572 patch_prober.go:28] interesting pod/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Jul 30 00:41:38.162528 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:38.162383 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" podUID="7589aa8b-bf6b-47e7-ac1b-926de232c2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Jul 30 00:41:39.161472 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:39.161434 2572 patch_prober.go:28] interesting pod/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Jul 30 00:41:39.161653 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:39.161493 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" podUID="7589aa8b-bf6b-47e7-ac1b-926de232c2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Jul 30 00:41:40.161942 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:40.161905 2572 patch_prober.go:28] interesting pod/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Jul 30 00:41:40.162338 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:40.161955 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" podUID="7589aa8b-bf6b-47e7-ac1b-926de232c2e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Jul 30 00:41:41.165981 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:41.165951 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:41.166358 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:41.166292 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:41.166912 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:41.166893 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:41:46.866445 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:46.866412 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:41:47.201336 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.201246 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:41:47.469799 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.469718 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx"] Jul 30 00:41:47.470070 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.470057 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0ba13ec-f05e-41b5-a745-5b7413ea158c" containerName="istio-proxy" Jul 30 00:41:47.472084 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.472065 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.474456 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.474439 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5f678c-openshift-default-dockercfg-rc5m4\"" Jul 30 00:41:47.484455 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.484435 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx"] Jul 30 00:41:47.494508 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.494484 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:41:47.633577 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.633547 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwq5x\" (UniqueName: \"kubernetes.io/projected/8ad17db3-9ef4-419d-a063-c91bed7146b2-kube-api-access-gwq5x\") pod \"e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.633758 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.633598 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-istio-envoy\") pod \"e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.633758 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.633676 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-workload-socket\") pod \"e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.633758 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.633723 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad17db3-9ef4-419d-a063-c91bed7146b2-istio-podinfo\") pod \"e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.633894 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.633792 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-workload-certs\") pod \"e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.633894 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.633817 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad17db3-9ef4-419d-a063-c91bed7146b2-istio-token\") pod \"e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.633894 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.633836 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad17db3-9ef4-419d-a063-c91bed7146b2-istiod-ca-cert\") pod \"e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.633894 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.633858 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-istio-data\") pod \"e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.634027 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.633900 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-credential-socket\") pod \"e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.735369 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.735284 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-workload-socket\") pod \"e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.735369 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.735334 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad17db3-9ef4-419d-a063-c91bed7146b2-istio-podinfo\") pod \"e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.735599 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.735458 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-workload-certs\") pod \"e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.735599 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.735496 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad17db3-9ef4-419d-a063-c91bed7146b2-istio-token\") pod \"e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.735599 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.735518 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad17db3-9ef4-419d-a063-c91bed7146b2-istiod-ca-cert\") pod \"e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.735599 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.735554 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-istio-data\") pod \"e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.735794 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.735597 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-credential-socket\") pod \"e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.735794 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.735655 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwq5x\" (UniqueName: \"kubernetes.io/projected/8ad17db3-9ef4-419d-a063-c91bed7146b2-kube-api-access-gwq5x\") pod \"e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.735794 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.735696 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-istio-envoy\") pod \"e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.735794 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.735735 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-workload-socket\") pod \"e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.736003 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.735980 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-credential-socket\") pod \"e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.736136 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.736006 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-istio-data\") pod \"e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.736136 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.736086 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-workload-certs\") pod \"e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.736232 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.736208 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad17db3-9ef4-419d-a063-c91bed7146b2-istiod-ca-cert\") pod \"e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.737861 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.737841 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-istio-envoy\") pod \"e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.737957 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.737940 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad17db3-9ef4-419d-a063-c91bed7146b2-istio-podinfo\") pod \"e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.742979 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.742957 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad17db3-9ef4-419d-a063-c91bed7146b2-istio-token\") pod \"e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.743376 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.743354 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwq5x\" (UniqueName: \"kubernetes.io/projected/8ad17db3-9ef4-419d-a063-c91bed7146b2-kube-api-access-gwq5x\") pod \"e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.781752 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.781722 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:47.785969 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.785941 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:41:47.921298 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.921275 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx"] Jul 30 00:41:47.923835 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:41:47.923808 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad17db3_9ef4_419d_a063_c91bed7146b2.slice/crio-16026477afd76fe07af29cb952d589ad5e86f65250eeed2ed1cfd5f952274363 WatchSource:0}: Error finding container 16026477afd76fe07af29cb952d589ad5e86f65250eeed2ed1cfd5f952274363: Status 404 returned error can't find the container with id 16026477afd76fe07af29cb952d589ad5e86f65250eeed2ed1cfd5f952274363 Jul 30 00:41:47.926437 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.926408 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:41:47.926526 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.926470 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:41:47.926526 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:47.926500 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:41:48.058176 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:48.058068 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" event={"ID":"8ad17db3-9ef4-419d-a063-c91bed7146b2","Type":"ContainerStarted","Data":"a072e15df27ca79a86bf27416ee6dd29fabe3d0a709a930943968313e40a54b2"} Jul 30 00:41:48.058176 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:48.058140 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" event={"ID":"8ad17db3-9ef4-419d-a063-c91bed7146b2","Type":"ContainerStarted","Data":"16026477afd76fe07af29cb952d589ad5e86f65250eeed2ed1cfd5f952274363"} Jul 30 00:41:48.078334 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:48.078284 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" podStartSLOduration=1.078267612 podStartE2EDuration="1.078267612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:41:48.077435825 +0000 UTC m=+2395.580694086" watchObservedRunningTime="2026-07-30 00:41:48.078267612 +0000 UTC m=+2395.581525946" Jul 30 00:41:48.784273 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:48.784230 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:48.789530 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:48.789501 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:49.062496 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:49.062420 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:49.063410 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:49.063392 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:41:49.975160 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:49.975129 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:41:50.073397 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:50.073362 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:41:51.215748 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:51.215711 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:41:51.313622 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:51.313590 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:41:51.759315 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:51.759286 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:41:51.853490 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:41:51.853453 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:42:04.391859 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:42:04.391824 2572 patch_prober.go:28] interesting pod/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": read tcp 10.133.0.2:39786->10.133.0.33:15021: read: connection reset by peer" start-of-body= Jul 30 00:42:04.392256 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:42:04.391911 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" podUID="8ad17db3-9ef4-419d-a063-c91bed7146b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": read tcp 10.133.0.2:39786->10.133.0.33:15021: read: connection reset by peer" Jul 30 00:42:05.132228 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:42:05.132196 2572 generic.go:356] "Generic (PLEG): container finished" podID="8ad17db3-9ef4-419d-a063-c91bed7146b2" containerID="a072e15df27ca79a86bf27416ee6dd29fabe3d0a709a930943968313e40a54b2" exitCode=0 Jul 30 00:42:05.132427 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:42:05.132277 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" event={"ID":"8ad17db3-9ef4-419d-a063-c91bed7146b2","Type":"ContainerDied","Data":"a072e15df27ca79a86bf27416ee6dd29fabe3d0a709a930943968313e40a54b2"} Jul 30 00:42:05.132705 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:42:05.132687 2572 scope.go:117] "RemoveContainer" containerID="a072e15df27ca79a86bf27416ee6dd29fabe3d0a709a930943968313e40a54b2" Jul 30 00:42:05.133691 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:42:05.133663 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:42:05.133773 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:42:05.133727 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:42:05.133773 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:42:05.133763 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:42:06.138463 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:42:06.138431 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" event={"ID":"8ad17db3-9ef4-419d-a063-c91bed7146b2","Type":"ContainerStarted","Data":"92654731dfa0b43122dfbb17db9fc87cc380d8007a43582540ac6dc6ec197737"} Jul 30 00:42:06.782904 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:42:06.782870 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:42:06.783142 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:42:06.783096 2572 patch_prober.go:28] interesting pod/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 30 00:42:06.783197 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:42:06.783179 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" podUID="8ad17db3-9ef4-419d-a063-c91bed7146b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 30 00:42:07.782825 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:42:07.782789 2572 patch_prober.go:28] interesting pod/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 30 00:42:07.783247 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:42:07.782845 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" podUID="8ad17db3-9ef4-419d-a063-c91bed7146b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 30 00:42:08.782350 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:42:08.782316 2572 patch_prober.go:28] interesting pod/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 30 00:42:08.782517 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:42:08.782367 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" podUID="8ad17db3-9ef4-419d-a063-c91bed7146b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 30 00:42:09.782995 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:42:09.782959 2572 patch_prober.go:28] interesting pod/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 30 00:42:09.783413 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:42:09.783017 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" podUID="8ad17db3-9ef4-419d-a063-c91bed7146b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 30 00:42:10.786282 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:42:10.786252 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:42:10.786663 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:42:10.786499 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:42:10.787098 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:42:10.787081 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:42:14.955573 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:42:14.955448 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:42:20.038986 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:42:20.038938 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:42:54.296037 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:42:54.295988 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:42:56.574150 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:42:56.574087 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:43:44.663054 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:43:44.663001 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:43:45.727740 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:43:45.727699 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:43:56.232387 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:43:56.232340 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:43:56.301861 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:43:56.301827 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:43:56.363443 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:43:56.363414 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:43:56.424667 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:43:56.424630 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:43:56.763303 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:43:56.763265 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:02.437267 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:02.437233 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:02.501664 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:02.501622 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:02.679878 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:02.679845 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd"] Jul 30 00:44:02.680129 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:02.680089 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" podUID="7589aa8b-bf6b-47e7-ac1b-926de232c2e4" containerName="istio-proxy" containerID="cri-o://3c9e24b9ce2737e0057acc1ae7b2850f7fea1dbedda6c9dd637491ac16cf0cb0" gracePeriod=30 Jul 30 00:44:02.704157 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:02.704077 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:02.952662 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:02.952625 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:07.927448 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:07.927423 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:44:08.086179 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.086151 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istiod-ca-cert\") pod \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " Jul 30 00:44:08.086179 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.086184 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-workload-socket\") pod \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " Jul 30 00:44:08.086424 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.086200 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-credential-socket\") pod \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " Jul 30 00:44:08.086424 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.086220 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-workload-certs\") pod \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " Jul 30 00:44:08.086424 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.086238 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istio-token\") pod \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " Jul 30 00:44:08.086424 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.086299 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istio-envoy\") pod \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " Jul 30 00:44:08.086424 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.086323 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrnnz\" (UniqueName: \"kubernetes.io/projected/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-kube-api-access-nrnnz\") pod \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " Jul 30 00:44:08.086424 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.086392 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istio-podinfo\") pod \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " Jul 30 00:44:08.086424 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.086416 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istio-data\") pod \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\" (UID: \"7589aa8b-bf6b-47e7-ac1b-926de232c2e4\") " Jul 30 00:44:08.086759 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.086503 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7589aa8b-bf6b-47e7-ac1b-926de232c2e4" (UID: "7589aa8b-bf6b-47e7-ac1b-926de232c2e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:44:08.086759 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.086566 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7589aa8b-bf6b-47e7-ac1b-926de232c2e4" (UID: "7589aa8b-bf6b-47e7-ac1b-926de232c2e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:44:08.086759 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.086601 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7589aa8b-bf6b-47e7-ac1b-926de232c2e4" (UID: "7589aa8b-bf6b-47e7-ac1b-926de232c2e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:44:08.086759 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.086608 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7589aa8b-bf6b-47e7-ac1b-926de232c2e4" (UID: "7589aa8b-bf6b-47e7-ac1b-926de232c2e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:44:08.086971 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.086807 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istiod-ca-cert\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:44:08.086971 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.086829 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-workload-socket\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:44:08.086971 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.086845 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-credential-socket\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:44:08.086971 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.086858 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-workload-certs\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:44:08.087474 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.087451 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istio-data" (OuterVolumeSpecName: "istio-data") pod "7589aa8b-bf6b-47e7-ac1b-926de232c2e4" (UID: "7589aa8b-bf6b-47e7-ac1b-926de232c2e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:44:08.088652 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.088630 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istio-token" (OuterVolumeSpecName: "istio-token") pod "7589aa8b-bf6b-47e7-ac1b-926de232c2e4" (UID: "7589aa8b-bf6b-47e7-ac1b-926de232c2e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:44:08.088812 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.088796 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7589aa8b-bf6b-47e7-ac1b-926de232c2e4" (UID: "7589aa8b-bf6b-47e7-ac1b-926de232c2e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 00:44:08.089047 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.089021 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7589aa8b-bf6b-47e7-ac1b-926de232c2e4" (UID: "7589aa8b-bf6b-47e7-ac1b-926de232c2e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:44:08.089157 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.089046 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-kube-api-access-nrnnz" (OuterVolumeSpecName: "kube-api-access-nrnnz") pod "7589aa8b-bf6b-47e7-ac1b-926de232c2e4" (UID: "7589aa8b-bf6b-47e7-ac1b-926de232c2e4"). InnerVolumeSpecName "kube-api-access-nrnnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:44:08.187302 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.187270 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istio-token\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:44:08.187302 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.187295 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istio-envoy\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:44:08.187302 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.187306 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrnnz\" (UniqueName: \"kubernetes.io/projected/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-kube-api-access-nrnnz\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:44:08.187526 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.187315 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istio-podinfo\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:44:08.187526 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.187324 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7589aa8b-bf6b-47e7-ac1b-926de232c2e4-istio-data\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:44:08.658841 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.658808 2572 generic.go:356] "Generic (PLEG): container finished" podID="7589aa8b-bf6b-47e7-ac1b-926de232c2e4" containerID="3c9e24b9ce2737e0057acc1ae7b2850f7fea1dbedda6c9dd637491ac16cf0cb0" exitCode=0 Jul 30 00:44:08.659055 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.658859 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" Jul 30 00:44:08.659055 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.658900 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" event={"ID":"7589aa8b-bf6b-47e7-ac1b-926de232c2e4","Type":"ContainerDied","Data":"3c9e24b9ce2737e0057acc1ae7b2850f7fea1dbedda6c9dd637491ac16cf0cb0"} Jul 30 00:44:08.659055 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.658954 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd" event={"ID":"7589aa8b-bf6b-47e7-ac1b-926de232c2e4","Type":"ContainerDied","Data":"1813e09342e792e7fea07deab625b9bde0a0fb9d5441e6ef0043d9685f58c6ae"} Jul 30 00:44:08.659055 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.658977 2572 scope.go:117] "RemoveContainer" containerID="3c9e24b9ce2737e0057acc1ae7b2850f7fea1dbedda6c9dd637491ac16cf0cb0" Jul 30 00:44:08.667669 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.667652 2572 scope.go:117] "RemoveContainer" containerID="d5059f746492a75cda8ed5d7a021afce46f5df1ed810651bdc2cb946002eb0b3" Jul 30 00:44:08.674869 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.674851 2572 scope.go:117] "RemoveContainer" containerID="3c9e24b9ce2737e0057acc1ae7b2850f7fea1dbedda6c9dd637491ac16cf0cb0" Jul 30 00:44:08.675128 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:44:08.675097 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c9e24b9ce2737e0057acc1ae7b2850f7fea1dbedda6c9dd637491ac16cf0cb0\": container with ID starting with 3c9e24b9ce2737e0057acc1ae7b2850f7fea1dbedda6c9dd637491ac16cf0cb0 not found: ID does not exist" containerID="3c9e24b9ce2737e0057acc1ae7b2850f7fea1dbedda6c9dd637491ac16cf0cb0" Jul 30 00:44:08.675189 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.675135 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c9e24b9ce2737e0057acc1ae7b2850f7fea1dbedda6c9dd637491ac16cf0cb0"} err="failed to get container status \"3c9e24b9ce2737e0057acc1ae7b2850f7fea1dbedda6c9dd637491ac16cf0cb0\": rpc error: code = NotFound desc = could not find container \"3c9e24b9ce2737e0057acc1ae7b2850f7fea1dbedda6c9dd637491ac16cf0cb0\": container with ID starting with 3c9e24b9ce2737e0057acc1ae7b2850f7fea1dbedda6c9dd637491ac16cf0cb0 not found: ID does not exist" Jul 30 00:44:08.675189 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.675153 2572 scope.go:117] "RemoveContainer" containerID="d5059f746492a75cda8ed5d7a021afce46f5df1ed810651bdc2cb946002eb0b3" Jul 30 00:44:08.675399 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:44:08.675380 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5059f746492a75cda8ed5d7a021afce46f5df1ed810651bdc2cb946002eb0b3\": container with ID starting with d5059f746492a75cda8ed5d7a021afce46f5df1ed810651bdc2cb946002eb0b3 not found: ID does not exist" containerID="d5059f746492a75cda8ed5d7a021afce46f5df1ed810651bdc2cb946002eb0b3" Jul 30 00:44:08.675451 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.675408 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5059f746492a75cda8ed5d7a021afce46f5df1ed810651bdc2cb946002eb0b3"} err="failed to get container status \"d5059f746492a75cda8ed5d7a021afce46f5df1ed810651bdc2cb946002eb0b3\": rpc error: code = NotFound desc = could not find container \"d5059f746492a75cda8ed5d7a021afce46f5df1ed810651bdc2cb946002eb0b3\": container with ID starting with d5059f746492a75cda8ed5d7a021afce46f5df1ed810651bdc2cb946002eb0b3 not found: ID does not exist" Jul 30 00:44:08.680432 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.680412 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd"] Jul 30 00:44:08.684125 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.684084 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d11ca7-openshift-default-7d96f7cb68-qtztd"] Jul 30 00:44:08.992372 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:08.992289 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7589aa8b-bf6b-47e7-ac1b-926de232c2e4" path="/var/lib/kubelet/pods/7589aa8b-bf6b-47e7-ac1b-926de232c2e4/volumes" Jul 30 00:44:13.430462 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:13.430422 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:13.530687 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:13.530655 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:13.594367 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:13.594334 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:13.937467 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:13.937438 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:19.760689 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:19.760657 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:19.820102 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:19.820073 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:20.003157 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:20.003122 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx"] Jul 30 00:44:20.003344 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:20.003326 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" podUID="8ad17db3-9ef4-419d-a063-c91bed7146b2" containerName="istio-proxy" containerID="cri-o://92654731dfa0b43122dfbb17db9fc87cc380d8007a43582540ac6dc6ec197737" gracePeriod=30 Jul 30 00:44:20.025978 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:20.025895 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:20.240477 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:20.240440 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:20.560461 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:20.560433 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:20.865035 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:20.864986 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:21.443735 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:21.443697 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:23.479045 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:23.479004 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:23.578582 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:23.578545 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:24.680147 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:24.680117 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5"] Jul 30 00:44:24.680576 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:24.680449 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7589aa8b-bf6b-47e7-ac1b-926de232c2e4" containerName="istio-proxy" Jul 30 00:44:24.680576 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:24.680460 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7589aa8b-bf6b-47e7-ac1b-926de232c2e4" containerName="istio-proxy" Jul 30 00:44:24.680576 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:24.680468 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7589aa8b-bf6b-47e7-ac1b-926de232c2e4" containerName="istio-proxy" Jul 30 00:44:24.680576 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:24.680474 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7589aa8b-bf6b-47e7-ac1b-926de232c2e4" containerName="istio-proxy" Jul 30 00:44:24.680576 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:24.680525 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7589aa8b-bf6b-47e7-ac1b-926de232c2e4" containerName="istio-proxy" Jul 30 00:44:24.680576 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:24.680534 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7589aa8b-bf6b-47e7-ac1b-926de232c2e4" containerName="istio-proxy" Jul 30 00:44:24.683547 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:24.683524 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5" Jul 30 00:44:24.686297 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:24.686272 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-dc78bf-serving-cert\"" Jul 30 00:44:24.693711 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:24.693685 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5"] Jul 30 00:44:24.717871 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:24.717845 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:24.722302 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:24.722279 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb8b57cc-a8b4-4a01-8777-b0ffe9712728-openshift-service-ca\") pod \"maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5\" (UID: \"fb8b57cc-a8b4-4a01-8777-b0ffe9712728\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5" Jul 30 00:44:24.722410 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:24.722333 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb8b57cc-a8b4-4a01-8777-b0ffe9712728-maas-api-tls\") pod \"maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5\" (UID: \"fb8b57cc-a8b4-4a01-8777-b0ffe9712728\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5" Jul 30 00:44:24.722410 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:24.722393 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxhtj\" (UniqueName: \"kubernetes.io/projected/fb8b57cc-a8b4-4a01-8777-b0ffe9712728-kube-api-access-qxhtj\") pod \"maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5\" (UID: \"fb8b57cc-a8b4-4a01-8777-b0ffe9712728\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5" Jul 30 00:44:24.823142 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:24.823092 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb8b57cc-a8b4-4a01-8777-b0ffe9712728-openshift-service-ca\") pod \"maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5\" (UID: \"fb8b57cc-a8b4-4a01-8777-b0ffe9712728\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5" Jul 30 00:44:24.823321 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:24.823168 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb8b57cc-a8b4-4a01-8777-b0ffe9712728-maas-api-tls\") pod \"maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5\" (UID: \"fb8b57cc-a8b4-4a01-8777-b0ffe9712728\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5" Jul 30 00:44:24.823321 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:24.823213 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxhtj\" (UniqueName: \"kubernetes.io/projected/fb8b57cc-a8b4-4a01-8777-b0ffe9712728-kube-api-access-qxhtj\") pod \"maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5\" (UID: \"fb8b57cc-a8b4-4a01-8777-b0ffe9712728\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5" Jul 30 00:44:24.823689 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:24.823670 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb8b57cc-a8b4-4a01-8777-b0ffe9712728-openshift-service-ca\") pod \"maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5\" (UID: \"fb8b57cc-a8b4-4a01-8777-b0ffe9712728\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5" Jul 30 00:44:24.825491 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:24.825473 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb8b57cc-a8b4-4a01-8777-b0ffe9712728-maas-api-tls\") pod \"maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5\" (UID: \"fb8b57cc-a8b4-4a01-8777-b0ffe9712728\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5" Jul 30 00:44:24.831446 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:24.831414 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxhtj\" (UniqueName: \"kubernetes.io/projected/fb8b57cc-a8b4-4a01-8777-b0ffe9712728-kube-api-access-qxhtj\") pod \"maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5\" (UID: \"fb8b57cc-a8b4-4a01-8777-b0ffe9712728\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5" Jul 30 00:44:24.838386 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:24.838361 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:24.995194 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:24.995099 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5" Jul 30 00:44:25.130639 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.130613 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5"] Jul 30 00:44:25.132917 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:44:25.132886 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb8b57cc_a8b4_4a01_8777_b0ffe9712728.slice/crio-42d9b99250a8d91b850418825906b6a3e049e4d46b40c648ff2265a9e586426c WatchSource:0}: Error finding container 42d9b99250a8d91b850418825906b6a3e049e4d46b40c648ff2265a9e586426c: Status 404 returned error can't find the container with id 42d9b99250a8d91b850418825906b6a3e049e4d46b40c648ff2265a9e586426c Jul 30 00:44:25.134298 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.134280 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 00:44:25.176975 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.176600 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:25.237822 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.237798 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:44:25.264486 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.264413 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:25.328236 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.328204 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad17db3-9ef4-419d-a063-c91bed7146b2-istio-podinfo\") pod \"8ad17db3-9ef4-419d-a063-c91bed7146b2\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " Jul 30 00:44:25.328401 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.328245 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-istio-data\") pod \"8ad17db3-9ef4-419d-a063-c91bed7146b2\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " Jul 30 00:44:25.328401 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.328284 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwq5x\" (UniqueName: \"kubernetes.io/projected/8ad17db3-9ef4-419d-a063-c91bed7146b2-kube-api-access-gwq5x\") pod \"8ad17db3-9ef4-419d-a063-c91bed7146b2\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " Jul 30 00:44:25.328401 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.328311 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad17db3-9ef4-419d-a063-c91bed7146b2-istio-token\") pod \"8ad17db3-9ef4-419d-a063-c91bed7146b2\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " Jul 30 00:44:25.328401 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.328360 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-credential-socket\") pod \"8ad17db3-9ef4-419d-a063-c91bed7146b2\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " Jul 30 00:44:25.328608 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.328409 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-workload-certs\") pod \"8ad17db3-9ef4-419d-a063-c91bed7146b2\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " Jul 30 00:44:25.328608 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.328436 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-istio-envoy\") pod \"8ad17db3-9ef4-419d-a063-c91bed7146b2\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " Jul 30 00:44:25.328608 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.328486 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-workload-socket\") pod \"8ad17db3-9ef4-419d-a063-c91bed7146b2\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " Jul 30 00:44:25.328608 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.328510 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad17db3-9ef4-419d-a063-c91bed7146b2-istiod-ca-cert\") pod \"8ad17db3-9ef4-419d-a063-c91bed7146b2\" (UID: \"8ad17db3-9ef4-419d-a063-c91bed7146b2\") " Jul 30 00:44:25.328816 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.328707 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8ad17db3-9ef4-419d-a063-c91bed7146b2" (UID: "8ad17db3-9ef4-419d-a063-c91bed7146b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:44:25.328816 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.328724 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8ad17db3-9ef4-419d-a063-c91bed7146b2" (UID: "8ad17db3-9ef4-419d-a063-c91bed7146b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:44:25.328922 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.328830 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-credential-socket\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:44:25.328922 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.328848 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-workload-certs\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:44:25.329023 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.328937 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8ad17db3-9ef4-419d-a063-c91bed7146b2" (UID: "8ad17db3-9ef4-419d-a063-c91bed7146b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:44:25.329023 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.329009 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ad17db3-9ef4-419d-a063-c91bed7146b2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8ad17db3-9ef4-419d-a063-c91bed7146b2" (UID: "8ad17db3-9ef4-419d-a063-c91bed7146b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:44:25.329304 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.329286 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-istio-data" (OuterVolumeSpecName: "istio-data") pod "8ad17db3-9ef4-419d-a063-c91bed7146b2" (UID: "8ad17db3-9ef4-419d-a063-c91bed7146b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:44:25.331049 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.331013 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8ad17db3-9ef4-419d-a063-c91bed7146b2" (UID: "8ad17db3-9ef4-419d-a063-c91bed7146b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:44:25.331049 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.331027 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad17db3-9ef4-419d-a063-c91bed7146b2-istio-token" (OuterVolumeSpecName: "istio-token") pod "8ad17db3-9ef4-419d-a063-c91bed7146b2" (UID: "8ad17db3-9ef4-419d-a063-c91bed7146b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:44:25.331049 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.331043 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ad17db3-9ef4-419d-a063-c91bed7146b2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8ad17db3-9ef4-419d-a063-c91bed7146b2" (UID: "8ad17db3-9ef4-419d-a063-c91bed7146b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 00:44:25.331245 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.331061 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad17db3-9ef4-419d-a063-c91bed7146b2-kube-api-access-gwq5x" (OuterVolumeSpecName: "kube-api-access-gwq5x") pod "8ad17db3-9ef4-419d-a063-c91bed7146b2" (UID: "8ad17db3-9ef4-419d-a063-c91bed7146b2"). InnerVolumeSpecName "kube-api-access-gwq5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:44:25.429542 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.429508 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-istio-envoy\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:44:25.429542 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.429535 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-workload-socket\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:44:25.429542 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.429545 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ad17db3-9ef4-419d-a063-c91bed7146b2-istiod-ca-cert\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:44:25.429771 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.429553 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ad17db3-9ef4-419d-a063-c91bed7146b2-istio-podinfo\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:44:25.429771 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.429562 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ad17db3-9ef4-419d-a063-c91bed7146b2-istio-data\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:44:25.429771 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.429570 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwq5x\" (UniqueName: \"kubernetes.io/projected/8ad17db3-9ef4-419d-a063-c91bed7146b2-kube-api-access-gwq5x\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:44:25.429771 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.429581 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ad17db3-9ef4-419d-a063-c91bed7146b2-istio-token\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:44:25.728626 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.728588 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5" event={"ID":"fb8b57cc-a8b4-4a01-8777-b0ffe9712728","Type":"ContainerStarted","Data":"c34f4fca2aafcd4b30c85bf15b81632a5089846963395d5af416d4e4be22daa2"} Jul 30 00:44:25.729033 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.728646 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5" event={"ID":"fb8b57cc-a8b4-4a01-8777-b0ffe9712728","Type":"ContainerStarted","Data":"42d9b99250a8d91b850418825906b6a3e049e4d46b40c648ff2265a9e586426c"} Jul 30 00:44:25.729033 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.728702 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5" Jul 30 00:44:25.730397 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.730368 2572 generic.go:356] "Generic (PLEG): container finished" podID="8ad17db3-9ef4-419d-a063-c91bed7146b2" containerID="92654731dfa0b43122dfbb17db9fc87cc380d8007a43582540ac6dc6ec197737" exitCode=0 Jul 30 00:44:25.730537 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.730416 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" Jul 30 00:44:25.730537 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.730441 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" event={"ID":"8ad17db3-9ef4-419d-a063-c91bed7146b2","Type":"ContainerDied","Data":"92654731dfa0b43122dfbb17db9fc87cc380d8007a43582540ac6dc6ec197737"} Jul 30 00:44:25.730537 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.730486 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx" event={"ID":"8ad17db3-9ef4-419d-a063-c91bed7146b2","Type":"ContainerDied","Data":"16026477afd76fe07af29cb952d589ad5e86f65250eeed2ed1cfd5f952274363"} Jul 30 00:44:25.730537 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.730516 2572 scope.go:117] "RemoveContainer" containerID="92654731dfa0b43122dfbb17db9fc87cc380d8007a43582540ac6dc6ec197737" Jul 30 00:44:25.740811 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.740467 2572 scope.go:117] "RemoveContainer" containerID="a072e15df27ca79a86bf27416ee6dd29fabe3d0a709a930943968313e40a54b2" Jul 30 00:44:25.747975 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.747910 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5" podStartSLOduration=1.298844087 podStartE2EDuration="1.747891725s" totalImagesPullingTime="449.047638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:44:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:44:25.745324642 +0000 UTC m=+2553.248582913" watchObservedRunningTime="2026-07-30 00:44:25.747891725 +0000 UTC m=+2553.251149989" Jul 30 00:44:25.749317 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.749300 2572 scope.go:117] "RemoveContainer" containerID="92654731dfa0b43122dfbb17db9fc87cc380d8007a43582540ac6dc6ec197737" Jul 30 00:44:25.749576 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:44:25.749559 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92654731dfa0b43122dfbb17db9fc87cc380d8007a43582540ac6dc6ec197737\": container with ID starting with 92654731dfa0b43122dfbb17db9fc87cc380d8007a43582540ac6dc6ec197737 not found: ID does not exist" containerID="92654731dfa0b43122dfbb17db9fc87cc380d8007a43582540ac6dc6ec197737" Jul 30 00:44:25.749625 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.749585 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92654731dfa0b43122dfbb17db9fc87cc380d8007a43582540ac6dc6ec197737"} err="failed to get container status \"92654731dfa0b43122dfbb17db9fc87cc380d8007a43582540ac6dc6ec197737\": rpc error: code = NotFound desc = could not find container \"92654731dfa0b43122dfbb17db9fc87cc380d8007a43582540ac6dc6ec197737\": container with ID starting with 92654731dfa0b43122dfbb17db9fc87cc380d8007a43582540ac6dc6ec197737 not found: ID does not exist" Jul 30 00:44:25.749625 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.749601 2572 scope.go:117] "RemoveContainer" containerID="a072e15df27ca79a86bf27416ee6dd29fabe3d0a709a930943968313e40a54b2" Jul 30 00:44:25.749831 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:44:25.749817 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a072e15df27ca79a86bf27416ee6dd29fabe3d0a709a930943968313e40a54b2\": container with ID starting with a072e15df27ca79a86bf27416ee6dd29fabe3d0a709a930943968313e40a54b2 not found: ID does not exist" containerID="a072e15df27ca79a86bf27416ee6dd29fabe3d0a709a930943968313e40a54b2" Jul 30 00:44:25.749867 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.749836 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a072e15df27ca79a86bf27416ee6dd29fabe3d0a709a930943968313e40a54b2"} err="failed to get container status \"a072e15df27ca79a86bf27416ee6dd29fabe3d0a709a930943968313e40a54b2\": rpc error: code = NotFound desc = could not find container \"a072e15df27ca79a86bf27416ee6dd29fabe3d0a709a930943968313e40a54b2\": container with ID starting with a072e15df27ca79a86bf27416ee6dd29fabe3d0a709a930943968313e40a54b2 not found: ID does not exist" Jul 30 00:44:25.758398 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.758373 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx"] Jul 30 00:44:25.761707 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:25.761687 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5f678c-openshift-default-78b4b8849d-4lvmx"] Jul 30 00:44:26.992719 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:26.992688 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ad17db3-9ef4-419d-a063-c91bed7146b2" path="/var/lib/kubelet/pods/8ad17db3-9ef4-419d-a063-c91bed7146b2/volumes" Jul 30 00:44:31.741076 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:31.741044 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5" Jul 30 00:44:48.463074 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:48.463034 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:48.908884 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:48.908842 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:49.197430 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.197335 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w"] Jul 30 00:44:49.197766 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.197741 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ad17db3-9ef4-419d-a063-c91bed7146b2" containerName="istio-proxy" Jul 30 00:44:49.197766 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.197765 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ad17db3-9ef4-419d-a063-c91bed7146b2" containerName="istio-proxy" Jul 30 00:44:49.197971 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.197859 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ad17db3-9ef4-419d-a063-c91bed7146b2" containerName="istio-proxy" Jul 30 00:44:49.197971 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.197871 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ad17db3-9ef4-419d-a063-c91bed7146b2" containerName="istio-proxy" Jul 30 00:44:49.201207 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.201190 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.203810 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.203788 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-fb2e9f-openshift-default-dockercfg-ns4kq\"" Jul 30 00:44:49.211782 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.211762 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w"] Jul 30 00:44:49.224451 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.224428 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:49.332039 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.332005 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-credential-socket\") pod \"e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.332039 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.332042 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-workload-certs\") pod \"e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.332276 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.332066 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-istio-data\") pod \"e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.332276 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.332156 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-istio-envoy\") pod \"e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.332276 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.332215 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmx4n\" (UniqueName: \"kubernetes.io/projected/20ca2a62-7955-4174-9377-1d11d31728c9-kube-api-access-gmx4n\") pod \"e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.332276 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.332241 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20ca2a62-7955-4174-9377-1d11d31728c9-istio-token\") pod \"e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.332413 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.332279 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ca2a62-7955-4174-9377-1d11d31728c9-istiod-ca-cert\") pod \"e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.332413 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.332348 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-workload-socket\") pod \"e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.332413 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.332381 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ca2a62-7955-4174-9377-1d11d31728c9-istio-podinfo\") pod \"e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.433381 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.433337 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20ca2a62-7955-4174-9377-1d11d31728c9-istio-token\") pod \"e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.433568 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.433393 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ca2a62-7955-4174-9377-1d11d31728c9-istiod-ca-cert\") pod \"e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.433568 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.433418 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-workload-socket\") pod \"e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.433568 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.433439 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ca2a62-7955-4174-9377-1d11d31728c9-istio-podinfo\") pod \"e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.433568 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.433462 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-credential-socket\") pod \"e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.433568 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.433481 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-workload-certs\") pod \"e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.433568 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.433506 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-istio-data\") pod \"e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.433872 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.433716 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-istio-envoy\") pod \"e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.433872 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.433785 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmx4n\" (UniqueName: \"kubernetes.io/projected/20ca2a62-7955-4174-9377-1d11d31728c9-kube-api-access-gmx4n\") pod \"e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.433982 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.433883 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-workload-socket\") pod \"e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.433982 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.433941 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-istio-data\") pod \"e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.434091 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.434032 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-credential-socket\") pod \"e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.434189 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.434170 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-workload-certs\") pod \"e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.434378 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.434359 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ca2a62-7955-4174-9377-1d11d31728c9-istiod-ca-cert\") pod \"e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.435881 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.435860 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-istio-envoy\") pod \"e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.435881 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.435876 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ca2a62-7955-4174-9377-1d11d31728c9-istio-podinfo\") pod \"e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.442402 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.442364 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20ca2a62-7955-4174-9377-1d11d31728c9-istio-token\") pod \"e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.442519 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.442495 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmx4n\" (UniqueName: \"kubernetes.io/projected/20ca2a62-7955-4174-9377-1d11d31728c9-kube-api-access-gmx4n\") pod \"e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.511506 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.511412 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:49.658304 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.658278 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w"] Jul 30 00:44:49.663093 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:44:49.662601 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ca2a62_7955_4174_9377_1d11d31728c9.slice/crio-88ba2816c11e71984163c03f58815ea3f13d52148921247e78a6edb58b587aca WatchSource:0}: Error finding container 88ba2816c11e71984163c03f58815ea3f13d52148921247e78a6edb58b587aca: Status 404 returned error can't find the container with id 88ba2816c11e71984163c03f58815ea3f13d52148921247e78a6edb58b587aca Jul 30 00:44:49.664856 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.664824 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:44:49.664956 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.664894 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:44:49.664956 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.664923 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:44:49.699811 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.699782 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:49.832306 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.832274 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" event={"ID":"20ca2a62-7955-4174-9377-1d11d31728c9","Type":"ContainerStarted","Data":"5f5853365df5e73751b30cd5308e56d99145f7071c994c65b41cfe5a2de5e881"} Jul 30 00:44:49.832521 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.832327 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" event={"ID":"20ca2a62-7955-4174-9377-1d11d31728c9","Type":"ContainerStarted","Data":"88ba2816c11e71984163c03f58815ea3f13d52148921247e78a6edb58b587aca"} Jul 30 00:44:49.854345 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:49.854286 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" podStartSLOduration=0.854270237 podStartE2EDuration="854.270237ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:44:49.852673752 +0000 UTC m=+2577.355932014" watchObservedRunningTime="2026-07-30 00:44:49.854270237 +0000 UTC m=+2577.357528489" Jul 30 00:44:50.512148 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:50.512092 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:50.516824 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:50.516801 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:50.836503 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:50.836469 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:50.837318 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:50.837293 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:44:51.749070 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:51.749032 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:51.843591 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:51.843555 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:52.834372 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:52.834342 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5"] Jul 30 00:44:52.834834 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:52.834643 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ad17db3-9ef4-419d-a063-c91bed7146b2" containerName="istio-proxy" Jul 30 00:44:52.834834 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:52.834654 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ad17db3-9ef4-419d-a063-c91bed7146b2" containerName="istio-proxy" Jul 30 00:44:52.837731 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:52.837708 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5" Jul 30 00:44:52.840046 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:52.840012 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-fb2e9f-serving-cert\"" Jul 30 00:44:52.847853 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:52.847830 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5"] Jul 30 00:44:52.865686 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:52.865659 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:52.965925 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:52.965888 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjs5q\" (UniqueName: \"kubernetes.io/projected/039691ed-4e6d-4731-ba02-9403636aab90-kube-api-access-zjs5q\") pod \"maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5\" (UID: \"039691ed-4e6d-4731-ba02-9403636aab90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5" Jul 30 00:44:52.966149 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:52.966076 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/039691ed-4e6d-4731-ba02-9403636aab90-openshift-service-ca\") pod \"maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5\" (UID: \"039691ed-4e6d-4731-ba02-9403636aab90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5" Jul 30 00:44:52.966149 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:52.966095 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/039691ed-4e6d-4731-ba02-9403636aab90-maas-api-tls\") pod \"maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5\" (UID: \"039691ed-4e6d-4731-ba02-9403636aab90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5" Jul 30 00:44:53.067128 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:53.067069 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjs5q\" (UniqueName: \"kubernetes.io/projected/039691ed-4e6d-4731-ba02-9403636aab90-kube-api-access-zjs5q\") pod \"maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5\" (UID: \"039691ed-4e6d-4731-ba02-9403636aab90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5" Jul 30 00:44:53.067304 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:53.067181 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/039691ed-4e6d-4731-ba02-9403636aab90-openshift-service-ca\") pod \"maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5\" (UID: \"039691ed-4e6d-4731-ba02-9403636aab90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5" Jul 30 00:44:53.067304 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:53.067202 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/039691ed-4e6d-4731-ba02-9403636aab90-maas-api-tls\") pod \"maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5\" (UID: \"039691ed-4e6d-4731-ba02-9403636aab90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5" Jul 30 00:44:53.067858 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:53.067837 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/039691ed-4e6d-4731-ba02-9403636aab90-openshift-service-ca\") pod \"maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5\" (UID: \"039691ed-4e6d-4731-ba02-9403636aab90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5" Jul 30 00:44:53.069662 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:53.069645 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-fb2e9f-serving-cert\"" Jul 30 00:44:53.075285 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:53.075268 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjs5q\" (UniqueName: \"kubernetes.io/projected/039691ed-4e6d-4731-ba02-9403636aab90-kube-api-access-zjs5q\") pod \"maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5\" (UID: \"039691ed-4e6d-4731-ba02-9403636aab90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5" Jul 30 00:44:53.078036 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:44:53.078022 2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-serving-cert: secret "maas-api-e2e-api-b-fb2e9f-serving-cert" not found Jul 30 00:44:53.078093 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:44:53.078069 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/039691ed-4e6d-4731-ba02-9403636aab90-maas-api-tls podName:039691ed-4e6d-4731-ba02-9403636aab90 nodeName:}" failed. No retries permitted until 2026-07-30 00:44:53.578054953 +0000 UTC m=+2581.081313191 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/039691ed-4e6d-4731-ba02-9403636aab90-maas-api-tls") pod "maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5" (UID: "039691ed-4e6d-4731-ba02-9403636aab90") : secret "maas-api-e2e-api-b-fb2e9f-serving-cert" not found Jul 30 00:44:53.322808 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:53.322743 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:53.457018 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:53.456932 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:53.560784 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:53.560734 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:44:53.671675 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:53.671643 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/039691ed-4e6d-4731-ba02-9403636aab90-maas-api-tls\") pod \"maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5\" (UID: \"039691ed-4e6d-4731-ba02-9403636aab90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5" Jul 30 00:44:53.674373 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:53.674345 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/039691ed-4e6d-4731-ba02-9403636aab90-maas-api-tls\") pod \"maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5\" (UID: \"039691ed-4e6d-4731-ba02-9403636aab90\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5" Jul 30 00:44:53.750364 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:53.750279 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5" Jul 30 00:44:53.880938 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:53.880907 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5"] Jul 30 00:44:53.884145 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:44:53.884085 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod039691ed_4e6d_4731_ba02_9403636aab90.slice/crio-d57ab5580cc6912e5af1db66d2ba0dac8ff2a371cf79b992ee6de9d07ff78d0d WatchSource:0}: Error finding container d57ab5580cc6912e5af1db66d2ba0dac8ff2a371cf79b992ee6de9d07ff78d0d: Status 404 returned error can't find the container with id d57ab5580cc6912e5af1db66d2ba0dac8ff2a371cf79b992ee6de9d07ff78d0d Jul 30 00:44:54.855178 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:54.855141 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5" event={"ID":"039691ed-4e6d-4731-ba02-9403636aab90","Type":"ContainerStarted","Data":"3ff8ba8a765cf475aea06503b670ab5c98e601e36a438796c69697704c3b92ed"} Jul 30 00:44:54.855347 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:54.855225 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5" event={"ID":"039691ed-4e6d-4731-ba02-9403636aab90","Type":"ContainerStarted","Data":"d57ab5580cc6912e5af1db66d2ba0dac8ff2a371cf79b992ee6de9d07ff78d0d"} Jul 30 00:44:54.855347 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:54.855280 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5" Jul 30 00:44:54.872309 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:44:54.872257 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5" podStartSLOduration=2.416876857 podStartE2EDuration="2.872241742s" totalImagesPullingTime="455.364885ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:44:54.870580484 +0000 UTC m=+2582.373838781" watchObservedRunningTime="2026-07-30 00:44:54.872241742 +0000 UTC m=+2582.375500003" Jul 30 00:45:00.864374 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:00.864347 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5" Jul 30 00:45:06.878275 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:06.878231 2572 patch_prober.go:28] interesting pod/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 30 00:45:06.878697 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:06.878304 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" podUID="20ca2a62-7955-4174-9377-1d11d31728c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 30 00:45:07.911613 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:07.911586 2572 generic.go:356] "Generic (PLEG): container finished" podID="20ca2a62-7955-4174-9377-1d11d31728c9" containerID="5f5853365df5e73751b30cd5308e56d99145f7071c994c65b41cfe5a2de5e881" exitCode=0 Jul 30 00:45:07.911948 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:07.911625 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" event={"ID":"20ca2a62-7955-4174-9377-1d11d31728c9","Type":"ContainerDied","Data":"5f5853365df5e73751b30cd5308e56d99145f7071c994c65b41cfe5a2de5e881"} Jul 30 00:45:07.912028 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:07.912014 2572 scope.go:117] "RemoveContainer" containerID="5f5853365df5e73751b30cd5308e56d99145f7071c994c65b41cfe5a2de5e881" Jul 30 00:45:07.912997 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:07.912969 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:45:07.913049 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:07.913037 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:45:07.913084 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:07.913068 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:45:08.917748 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:08.917719 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" event={"ID":"20ca2a62-7955-4174-9377-1d11d31728c9","Type":"ContainerStarted","Data":"b77501e975bb16ce15a84a66dfc07191c7a0f223a44efade62e509d54f3200ea"} Jul 30 00:45:09.512089 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:09.512048 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:45:09.512324 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:09.512300 2572 patch_prober.go:28] interesting pod/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 30 00:45:09.512377 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:09.512350 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" podUID="20ca2a62-7955-4174-9377-1d11d31728c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 30 00:45:10.511944 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:10.511904 2572 patch_prober.go:28] interesting pod/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 30 00:45:10.512354 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:10.511972 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" podUID="20ca2a62-7955-4174-9377-1d11d31728c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 30 00:45:11.512302 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:11.512268 2572 patch_prober.go:28] interesting pod/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 30 00:45:11.512675 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:11.512328 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" podUID="20ca2a62-7955-4174-9377-1d11d31728c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 30 00:45:12.512633 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:12.512591 2572 patch_prober.go:28] interesting pod/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 30 00:45:12.513015 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:12.512651 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" podUID="20ca2a62-7955-4174-9377-1d11d31728c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 30 00:45:13.515921 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:13.515890 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:45:13.516362 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:13.516149 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:45:13.516827 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:13.516805 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:45:16.765935 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:16.765839 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:45:35.753250 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:35.753220 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5"] Jul 30 00:45:35.753783 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:35.753445 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5" podUID="fb8b57cc-a8b4-4a01-8777-b0ffe9712728" containerName="maas-api" containerID="cri-o://c34f4fca2aafcd4b30c85bf15b81632a5089846963395d5af416d4e4be22daa2" gracePeriod=30 Jul 30 00:45:35.783479 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:35.783436 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:45:35.851894 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:35.851848 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:45:35.923552 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:35.923508 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:45:35.997357 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:35.997333 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5" Jul 30 00:45:36.020298 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:36.020218 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:45:36.028177 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:36.028149 2572 generic.go:356] "Generic (PLEG): container finished" podID="fb8b57cc-a8b4-4a01-8777-b0ffe9712728" containerID="c34f4fca2aafcd4b30c85bf15b81632a5089846963395d5af416d4e4be22daa2" exitCode=0 Jul 30 00:45:36.028327 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:36.028199 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5" Jul 30 00:45:36.028327 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:36.028237 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5" event={"ID":"fb8b57cc-a8b4-4a01-8777-b0ffe9712728","Type":"ContainerDied","Data":"c34f4fca2aafcd4b30c85bf15b81632a5089846963395d5af416d4e4be22daa2"} Jul 30 00:45:36.028327 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:36.028299 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5" event={"ID":"fb8b57cc-a8b4-4a01-8777-b0ffe9712728","Type":"ContainerDied","Data":"42d9b99250a8d91b850418825906b6a3e049e4d46b40c648ff2265a9e586426c"} Jul 30 00:45:36.028477 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:36.028332 2572 scope.go:117] "RemoveContainer" containerID="c34f4fca2aafcd4b30c85bf15b81632a5089846963395d5af416d4e4be22daa2" Jul 30 00:45:36.036437 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:36.036409 2572 scope.go:117] "RemoveContainer" containerID="c34f4fca2aafcd4b30c85bf15b81632a5089846963395d5af416d4e4be22daa2" Jul 30 00:45:36.036751 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:45:36.036730 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c34f4fca2aafcd4b30c85bf15b81632a5089846963395d5af416d4e4be22daa2\": container with ID starting with c34f4fca2aafcd4b30c85bf15b81632a5089846963395d5af416d4e4be22daa2 not found: ID does not exist" containerID="c34f4fca2aafcd4b30c85bf15b81632a5089846963395d5af416d4e4be22daa2" Jul 30 00:45:36.036796 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:36.036761 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c34f4fca2aafcd4b30c85bf15b81632a5089846963395d5af416d4e4be22daa2"} err="failed to get container status \"c34f4fca2aafcd4b30c85bf15b81632a5089846963395d5af416d4e4be22daa2\": rpc error: code = NotFound desc = could not find container \"c34f4fca2aafcd4b30c85bf15b81632a5089846963395d5af416d4e4be22daa2\": container with ID starting with c34f4fca2aafcd4b30c85bf15b81632a5089846963395d5af416d4e4be22daa2 not found: ID does not exist" Jul 30 00:45:36.148281 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:36.148244 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxhtj\" (UniqueName: \"kubernetes.io/projected/fb8b57cc-a8b4-4a01-8777-b0ffe9712728-kube-api-access-qxhtj\") pod \"fb8b57cc-a8b4-4a01-8777-b0ffe9712728\" (UID: \"fb8b57cc-a8b4-4a01-8777-b0ffe9712728\") " Jul 30 00:45:36.148448 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:36.148383 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb8b57cc-a8b4-4a01-8777-b0ffe9712728-maas-api-tls\") pod \"fb8b57cc-a8b4-4a01-8777-b0ffe9712728\" (UID: \"fb8b57cc-a8b4-4a01-8777-b0ffe9712728\") " Jul 30 00:45:36.148448 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:36.148422 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb8b57cc-a8b4-4a01-8777-b0ffe9712728-openshift-service-ca\") pod \"fb8b57cc-a8b4-4a01-8777-b0ffe9712728\" (UID: \"fb8b57cc-a8b4-4a01-8777-b0ffe9712728\") " Jul 30 00:45:36.148834 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:36.148809 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb8b57cc-a8b4-4a01-8777-b0ffe9712728-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fb8b57cc-a8b4-4a01-8777-b0ffe9712728" (UID: "fb8b57cc-a8b4-4a01-8777-b0ffe9712728"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:45:36.150480 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:36.150450 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb8b57cc-a8b4-4a01-8777-b0ffe9712728-kube-api-access-qxhtj" (OuterVolumeSpecName: "kube-api-access-qxhtj") pod "fb8b57cc-a8b4-4a01-8777-b0ffe9712728" (UID: "fb8b57cc-a8b4-4a01-8777-b0ffe9712728"). InnerVolumeSpecName "kube-api-access-qxhtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:45:36.150611 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:36.150490 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb8b57cc-a8b4-4a01-8777-b0ffe9712728-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fb8b57cc-a8b4-4a01-8777-b0ffe9712728" (UID: "fb8b57cc-a8b4-4a01-8777-b0ffe9712728"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 00:45:36.249884 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:36.249851 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb8b57cc-a8b4-4a01-8777-b0ffe9712728-openshift-service-ca\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:45:36.249884 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:36.249879 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxhtj\" (UniqueName: \"kubernetes.io/projected/fb8b57cc-a8b4-4a01-8777-b0ffe9712728-kube-api-access-qxhtj\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:45:36.249884 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:36.249891 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb8b57cc-a8b4-4a01-8777-b0ffe9712728-maas-api-tls\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:45:36.318508 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:36.318421 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:45:36.349520 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:36.349490 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5"] Jul 30 00:45:36.355626 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:36.355605 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc78bf-76b8dc9f86-hd6r5"] Jul 30 00:45:36.992269 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:36.992228 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb8b57cc-a8b4-4a01-8777-b0ffe9712728" path="/var/lib/kubelet/pods/fb8b57cc-a8b4-4a01-8777-b0ffe9712728/volumes" Jul 30 00:45:42.039500 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:42.039460 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:45:42.102289 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:42.102245 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:45:42.326065 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:42.326027 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:45:42.531729 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:42.531691 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:45:58.033766 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:58.033733 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5"] Jul 30 00:45:58.034310 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:58.033943 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5" podUID="039691ed-4e6d-4731-ba02-9403636aab90" containerName="maas-api" containerID="cri-o://3ff8ba8a765cf475aea06503b670ab5c98e601e36a438796c69697704c3b92ed" gracePeriod=30 Jul 30 00:45:58.070021 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:58.069991 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:45:58.140805 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:58.140770 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:45:58.222705 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:58.222672 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:45:58.287445 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:58.287384 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5" Jul 30 00:45:58.327123 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:58.327079 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/039691ed-4e6d-4731-ba02-9403636aab90-maas-api-tls\") pod \"039691ed-4e6d-4731-ba02-9403636aab90\" (UID: \"039691ed-4e6d-4731-ba02-9403636aab90\") " Jul 30 00:45:58.327299 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:58.327134 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/039691ed-4e6d-4731-ba02-9403636aab90-openshift-service-ca\") pod \"039691ed-4e6d-4731-ba02-9403636aab90\" (UID: \"039691ed-4e6d-4731-ba02-9403636aab90\") " Jul 30 00:45:58.327299 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:58.327172 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjs5q\" (UniqueName: \"kubernetes.io/projected/039691ed-4e6d-4731-ba02-9403636aab90-kube-api-access-zjs5q\") pod \"039691ed-4e6d-4731-ba02-9403636aab90\" (UID: \"039691ed-4e6d-4731-ba02-9403636aab90\") " Jul 30 00:45:58.327503 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:58.327475 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/039691ed-4e6d-4731-ba02-9403636aab90-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "039691ed-4e6d-4731-ba02-9403636aab90" (UID: "039691ed-4e6d-4731-ba02-9403636aab90"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:45:58.329299 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:58.329270 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/039691ed-4e6d-4731-ba02-9403636aab90-kube-api-access-zjs5q" (OuterVolumeSpecName: "kube-api-access-zjs5q") pod "039691ed-4e6d-4731-ba02-9403636aab90" (UID: "039691ed-4e6d-4731-ba02-9403636aab90"). InnerVolumeSpecName "kube-api-access-zjs5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:45:58.329577 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:58.329554 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/039691ed-4e6d-4731-ba02-9403636aab90-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "039691ed-4e6d-4731-ba02-9403636aab90" (UID: "039691ed-4e6d-4731-ba02-9403636aab90"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 00:45:58.428174 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:58.428141 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/039691ed-4e6d-4731-ba02-9403636aab90-maas-api-tls\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:45:58.428174 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:58.428169 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/039691ed-4e6d-4731-ba02-9403636aab90-openshift-service-ca\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:45:58.428174 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:58.428179 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjs5q\" (UniqueName: \"kubernetes.io/projected/039691ed-4e6d-4731-ba02-9403636aab90-kube-api-access-zjs5q\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:45:58.606214 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:58.606180 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:45:59.124249 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:59.124212 2572 generic.go:356] "Generic (PLEG): container finished" podID="039691ed-4e6d-4731-ba02-9403636aab90" containerID="3ff8ba8a765cf475aea06503b670ab5c98e601e36a438796c69697704c3b92ed" exitCode=0 Jul 30 00:45:59.124643 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:59.124262 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5" Jul 30 00:45:59.124643 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:59.124298 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5" event={"ID":"039691ed-4e6d-4731-ba02-9403636aab90","Type":"ContainerDied","Data":"3ff8ba8a765cf475aea06503b670ab5c98e601e36a438796c69697704c3b92ed"} Jul 30 00:45:59.124643 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:59.124359 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5" event={"ID":"039691ed-4e6d-4731-ba02-9403636aab90","Type":"ContainerDied","Data":"d57ab5580cc6912e5af1db66d2ba0dac8ff2a371cf79b992ee6de9d07ff78d0d"} Jul 30 00:45:59.124643 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:59.124388 2572 scope.go:117] "RemoveContainer" containerID="3ff8ba8a765cf475aea06503b670ab5c98e601e36a438796c69697704c3b92ed" Jul 30 00:45:59.132499 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:59.132481 2572 scope.go:117] "RemoveContainer" containerID="3ff8ba8a765cf475aea06503b670ab5c98e601e36a438796c69697704c3b92ed" Jul 30 00:45:59.132765 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:45:59.132746 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ff8ba8a765cf475aea06503b670ab5c98e601e36a438796c69697704c3b92ed\": container with ID starting with 3ff8ba8a765cf475aea06503b670ab5c98e601e36a438796c69697704c3b92ed not found: ID does not exist" containerID="3ff8ba8a765cf475aea06503b670ab5c98e601e36a438796c69697704c3b92ed" Jul 30 00:45:59.132819 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:59.132775 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ff8ba8a765cf475aea06503b670ab5c98e601e36a438796c69697704c3b92ed"} err="failed to get container status \"3ff8ba8a765cf475aea06503b670ab5c98e601e36a438796c69697704c3b92ed\": rpc error: code = NotFound desc = could not find container \"3ff8ba8a765cf475aea06503b670ab5c98e601e36a438796c69697704c3b92ed\": container with ID starting with 3ff8ba8a765cf475aea06503b670ab5c98e601e36a438796c69697704c3b92ed not found: ID does not exist" Jul 30 00:45:59.142346 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:59.142323 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5"] Jul 30 00:45:59.148385 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:45:59.148364 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fb2e9f-7b5c68d9c7-m55b5"] Jul 30 00:46:00.992418 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:00.992386 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="039691ed-4e6d-4731-ba02-9403636aab90" path="/var/lib/kubelet/pods/039691ed-4e6d-4731-ba02-9403636aab90/volumes" Jul 30 00:46:04.349772 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:04.349737 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:46:04.543615 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:04.543577 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:46:04.577589 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:04.577554 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w"] Jul 30 00:46:04.577877 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:04.577854 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" podUID="20ca2a62-7955-4174-9377-1d11d31728c9" containerName="istio-proxy" containerID="cri-o://b77501e975bb16ce15a84a66dfc07191c7a0f223a44efade62e509d54f3200ea" gracePeriod=30 Jul 30 00:46:04.603538 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:04.603464 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:46:04.871868 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:04.871771 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:46:05.163152 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.163040 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:46:05.445282 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.445207 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv"] Jul 30 00:46:05.445640 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.445528 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb8b57cc-a8b4-4a01-8777-b0ffe9712728" containerName="maas-api" Jul 30 00:46:05.445640 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.445542 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb8b57cc-a8b4-4a01-8777-b0ffe9712728" containerName="maas-api" Jul 30 00:46:05.445640 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.445557 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="039691ed-4e6d-4731-ba02-9403636aab90" containerName="maas-api" Jul 30 00:46:05.445640 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.445564 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="039691ed-4e6d-4731-ba02-9403636aab90" containerName="maas-api" Jul 30 00:46:05.445640 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.445629 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="039691ed-4e6d-4731-ba02-9403636aab90" containerName="maas-api" Jul 30 00:46:05.445640 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.445637 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb8b57cc-a8b4-4a01-8777-b0ffe9712728" containerName="maas-api" Jul 30 00:46:05.448379 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.448359 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.450868 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.450851 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-4ef902-openshift-default-dockercfg-wgj78\"" Jul 30 00:46:05.461487 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.461463 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv"] Jul 30 00:46:05.489200 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.489168 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istio-podinfo\") pod \"e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.489356 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.489209 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-credential-socket\") pod \"e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.489356 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.489237 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istiod-ca-cert\") pod \"e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.489356 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.489256 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-workload-socket\") pod \"e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.489356 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.489284 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istio-token\") pod \"e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.489356 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.489347 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istio-data\") pod \"e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.489551 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.489380 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4l8z\" (UniqueName: \"kubernetes.io/projected/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-kube-api-access-l4l8z\") pod \"e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.489551 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.489408 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istio-envoy\") pod \"e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.489551 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.489525 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-workload-certs\") pod \"e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.493692 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.493669 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:46:05.590210 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.590169 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-credential-socket\") pod \"e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.590210 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.590213 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istiod-ca-cert\") pod \"e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.590472 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.590236 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-workload-socket\") pod \"e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.590472 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.590348 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istio-token\") pod \"e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.590472 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.590400 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istio-data\") pod \"e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.590472 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.590436 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4l8z\" (UniqueName: \"kubernetes.io/projected/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-kube-api-access-l4l8z\") pod \"e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.590472 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.590473 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istio-envoy\") pod \"e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.590741 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.590541 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-workload-certs\") pod \"e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.590741 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.590605 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istio-podinfo\") pod \"e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.590741 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.590622 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-credential-socket\") pod \"e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.590741 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.590648 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-workload-socket\") pod \"e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.590979 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.590946 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-workload-certs\") pod \"e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.591037 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.591010 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istio-data\") pod \"e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.591273 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.591248 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istiod-ca-cert\") pod \"e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.592788 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.592758 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istio-envoy\") pod \"e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.592979 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.592963 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istio-podinfo\") pod \"e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.598424 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.598401 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istio-token\") pod \"e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.598530 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.598495 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4l8z\" (UniqueName: \"kubernetes.io/projected/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-kube-api-access-l4l8z\") pod \"e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.616364 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.616337 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:46:05.758667 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.758570 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:05.889467 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.889440 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv"] Jul 30 00:46:05.891911 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:46:05.891880 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe8dcdb_21a5_45a2_8102_ab451b7172a7.slice/crio-95389ae5dc350ef7f0df66aff579584daa756a91eef26b6995537100f020a5f6 WatchSource:0}: Error finding container 95389ae5dc350ef7f0df66aff579584daa756a91eef26b6995537100f020a5f6: Status 404 returned error can't find the container with id 95389ae5dc350ef7f0df66aff579584daa756a91eef26b6995537100f020a5f6 Jul 30 00:46:05.894081 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.894053 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:46:05.894160 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.894137 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:46:05.894206 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:05.894168 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:46:06.154564 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:06.154530 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" event={"ID":"bbe8dcdb-21a5-45a2-8102-ab451b7172a7","Type":"ContainerStarted","Data":"713dfd5eef931190f025b5b2bf7907ef869eb9fbbbd482c229ec36b2ad7ec0ad"} Jul 30 00:46:06.154740 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:06.154582 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" event={"ID":"bbe8dcdb-21a5-45a2-8102-ab451b7172a7","Type":"ContainerStarted","Data":"95389ae5dc350ef7f0df66aff579584daa756a91eef26b6995537100f020a5f6"} Jul 30 00:46:06.177483 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:06.177434 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" podStartSLOduration=1.177416271 podStartE2EDuration="1.177416271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:46:06.176507816 +0000 UTC m=+2653.679766078" watchObservedRunningTime="2026-07-30 00:46:06.177416271 +0000 UTC m=+2653.680674530" Jul 30 00:46:06.759091 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:06.759061 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:06.764001 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:06.763975 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:07.159183 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:07.159150 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:07.160218 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:07.160197 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:08.053687 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:08.053653 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:46:08.157963 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:08.157928 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:46:09.295549 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:09.295517 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:46:09.429455 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:09.429424 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:46:09.729908 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:09.729874 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:46:09.825306 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:09.825281 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:46:09.841869 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:09.841817 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:46:09.929247 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:09.929216 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-istio-data\") pod \"20ca2a62-7955-4174-9377-1d11d31728c9\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " Jul 30 00:46:09.929437 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:09.929256 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ca2a62-7955-4174-9377-1d11d31728c9-istio-podinfo\") pod \"20ca2a62-7955-4174-9377-1d11d31728c9\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " Jul 30 00:46:09.929437 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:09.929285 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-workload-socket\") pod \"20ca2a62-7955-4174-9377-1d11d31728c9\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " Jul 30 00:46:09.929437 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:09.929348 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmx4n\" (UniqueName: \"kubernetes.io/projected/20ca2a62-7955-4174-9377-1d11d31728c9-kube-api-access-gmx4n\") pod \"20ca2a62-7955-4174-9377-1d11d31728c9\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " Jul 30 00:46:09.929437 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:09.929377 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-credential-socket\") pod \"20ca2a62-7955-4174-9377-1d11d31728c9\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " Jul 30 00:46:09.929437 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:09.929405 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-workload-certs\") pod \"20ca2a62-7955-4174-9377-1d11d31728c9\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " Jul 30 00:46:09.929707 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:09.929538 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-istio-envoy\") pod \"20ca2a62-7955-4174-9377-1d11d31728c9\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " Jul 30 00:46:09.929707 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:09.929595 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ca2a62-7955-4174-9377-1d11d31728c9-istiod-ca-cert\") pod \"20ca2a62-7955-4174-9377-1d11d31728c9\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " Jul 30 00:46:09.929707 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:09.929628 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20ca2a62-7955-4174-9377-1d11d31728c9-istio-token\") pod \"20ca2a62-7955-4174-9377-1d11d31728c9\" (UID: \"20ca2a62-7955-4174-9377-1d11d31728c9\") " Jul 30 00:46:09.929707 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:09.929643 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "20ca2a62-7955-4174-9377-1d11d31728c9" (UID: "20ca2a62-7955-4174-9377-1d11d31728c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:46:09.929926 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:09.929791 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "20ca2a62-7955-4174-9377-1d11d31728c9" (UID: "20ca2a62-7955-4174-9377-1d11d31728c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:46:09.930040 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:09.930014 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20ca2a62-7955-4174-9377-1d11d31728c9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "20ca2a62-7955-4174-9377-1d11d31728c9" (UID: "20ca2a62-7955-4174-9377-1d11d31728c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:46:09.930159 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:09.930134 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ca2a62-7955-4174-9377-1d11d31728c9-istiod-ca-cert\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:46:09.930159 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:09.930160 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-workload-socket\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:46:09.930357 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:09.930176 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-workload-certs\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:46:09.930357 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:09.930235 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "20ca2a62-7955-4174-9377-1d11d31728c9" (UID: "20ca2a62-7955-4174-9377-1d11d31728c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:46:09.930357 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:09.930284 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-istio-data" (OuterVolumeSpecName: "istio-data") pod "20ca2a62-7955-4174-9377-1d11d31728c9" (UID: "20ca2a62-7955-4174-9377-1d11d31728c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:46:09.931820 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:09.931796 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "20ca2a62-7955-4174-9377-1d11d31728c9" (UID: "20ca2a62-7955-4174-9377-1d11d31728c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:46:09.931914 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:09.931824 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ca2a62-7955-4174-9377-1d11d31728c9-kube-api-access-gmx4n" (OuterVolumeSpecName: "kube-api-access-gmx4n") pod "20ca2a62-7955-4174-9377-1d11d31728c9" (UID: "20ca2a62-7955-4174-9377-1d11d31728c9"). InnerVolumeSpecName "kube-api-access-gmx4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:46:09.932048 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:09.932026 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ca2a62-7955-4174-9377-1d11d31728c9-istio-token" (OuterVolumeSpecName: "istio-token") pod "20ca2a62-7955-4174-9377-1d11d31728c9" (UID: "20ca2a62-7955-4174-9377-1d11d31728c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:46:09.932246 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:09.932092 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20ca2a62-7955-4174-9377-1d11d31728c9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "20ca2a62-7955-4174-9377-1d11d31728c9" (UID: "20ca2a62-7955-4174-9377-1d11d31728c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 00:46:10.031597 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:10.031558 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmx4n\" (UniqueName: \"kubernetes.io/projected/20ca2a62-7955-4174-9377-1d11d31728c9-kube-api-access-gmx4n\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:46:10.031805 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:10.031689 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-credential-socket\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:46:10.031805 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:10.031707 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-istio-envoy\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:46:10.031805 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:10.031716 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20ca2a62-7955-4174-9377-1d11d31728c9-istio-token\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:46:10.031805 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:10.031724 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ca2a62-7955-4174-9377-1d11d31728c9-istio-data\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:46:10.031805 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:10.031733 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ca2a62-7955-4174-9377-1d11d31728c9-istio-podinfo\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:46:10.173600 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:10.173514 2572 generic.go:356] "Generic (PLEG): container finished" podID="20ca2a62-7955-4174-9377-1d11d31728c9" containerID="b77501e975bb16ce15a84a66dfc07191c7a0f223a44efade62e509d54f3200ea" exitCode=0 Jul 30 00:46:10.173600 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:10.173565 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" Jul 30 00:46:10.173779 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:10.173597 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" event={"ID":"20ca2a62-7955-4174-9377-1d11d31728c9","Type":"ContainerDied","Data":"b77501e975bb16ce15a84a66dfc07191c7a0f223a44efade62e509d54f3200ea"} Jul 30 00:46:10.173779 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:10.173648 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w" event={"ID":"20ca2a62-7955-4174-9377-1d11d31728c9","Type":"ContainerDied","Data":"88ba2816c11e71984163c03f58815ea3f13d52148921247e78a6edb58b587aca"} Jul 30 00:46:10.173779 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:10.173669 2572 scope.go:117] "RemoveContainer" containerID="b77501e975bb16ce15a84a66dfc07191c7a0f223a44efade62e509d54f3200ea" Jul 30 00:46:10.185374 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:10.185356 2572 scope.go:117] "RemoveContainer" containerID="5f5853365df5e73751b30cd5308e56d99145f7071c994c65b41cfe5a2de5e881" Jul 30 00:46:10.192880 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:10.192862 2572 scope.go:117] "RemoveContainer" containerID="b77501e975bb16ce15a84a66dfc07191c7a0f223a44efade62e509d54f3200ea" Jul 30 00:46:10.193160 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:46:10.193133 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b77501e975bb16ce15a84a66dfc07191c7a0f223a44efade62e509d54f3200ea\": container with ID starting with b77501e975bb16ce15a84a66dfc07191c7a0f223a44efade62e509d54f3200ea not found: ID does not exist" containerID="b77501e975bb16ce15a84a66dfc07191c7a0f223a44efade62e509d54f3200ea" Jul 30 00:46:10.193242 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:10.193171 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b77501e975bb16ce15a84a66dfc07191c7a0f223a44efade62e509d54f3200ea"} err="failed to get container status \"b77501e975bb16ce15a84a66dfc07191c7a0f223a44efade62e509d54f3200ea\": rpc error: code = NotFound desc = could not find container \"b77501e975bb16ce15a84a66dfc07191c7a0f223a44efade62e509d54f3200ea\": container with ID starting with b77501e975bb16ce15a84a66dfc07191c7a0f223a44efade62e509d54f3200ea not found: ID does not exist" Jul 30 00:46:10.193242 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:10.193193 2572 scope.go:117] "RemoveContainer" containerID="5f5853365df5e73751b30cd5308e56d99145f7071c994c65b41cfe5a2de5e881" Jul 30 00:46:10.193452 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:46:10.193433 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f5853365df5e73751b30cd5308e56d99145f7071c994c65b41cfe5a2de5e881\": container with ID starting with 5f5853365df5e73751b30cd5308e56d99145f7071c994c65b41cfe5a2de5e881 not found: ID does not exist" containerID="5f5853365df5e73751b30cd5308e56d99145f7071c994c65b41cfe5a2de5e881" Jul 30 00:46:10.193505 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:10.193457 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f5853365df5e73751b30cd5308e56d99145f7071c994c65b41cfe5a2de5e881"} err="failed to get container status \"5f5853365df5e73751b30cd5308e56d99145f7071c994c65b41cfe5a2de5e881\": rpc error: code = NotFound desc = could not find container \"5f5853365df5e73751b30cd5308e56d99145f7071c994c65b41cfe5a2de5e881\": container with ID starting with 5f5853365df5e73751b30cd5308e56d99145f7071c994c65b41cfe5a2de5e881 not found: ID does not exist" Jul 30 00:46:10.196461 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:10.196441 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w"] Jul 30 00:46:10.201435 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:10.201413 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-fb2e9f-openshift-default-5446ff7c5d-k477w"] Jul 30 00:46:10.993029 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:10.992989 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20ca2a62-7955-4174-9377-1d11d31728c9" path="/var/lib/kubelet/pods/20ca2a62-7955-4174-9377-1d11d31728c9/volumes" Jul 30 00:46:22.160493 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:22.160454 2572 patch_prober.go:28] interesting pod/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 30 00:46:22.160913 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:22.160523 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" podUID="bbe8dcdb-21a5-45a2-8102-ab451b7172a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 30 00:46:22.225882 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:22.225851 2572 generic.go:356] "Generic (PLEG): container finished" podID="bbe8dcdb-21a5-45a2-8102-ab451b7172a7" containerID="713dfd5eef931190f025b5b2bf7907ef869eb9fbbbd482c229ec36b2ad7ec0ad" exitCode=0 Jul 30 00:46:22.226042 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:22.225925 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" event={"ID":"bbe8dcdb-21a5-45a2-8102-ab451b7172a7","Type":"ContainerDied","Data":"713dfd5eef931190f025b5b2bf7907ef869eb9fbbbd482c229ec36b2ad7ec0ad"} Jul 30 00:46:22.226335 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:22.226320 2572 scope.go:117] "RemoveContainer" containerID="713dfd5eef931190f025b5b2bf7907ef869eb9fbbbd482c229ec36b2ad7ec0ad" Jul 30 00:46:22.227290 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:22.227264 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:46:22.227357 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:22.227330 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:46:22.227399 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:22.227361 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:46:23.232663 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:23.232610 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" event={"ID":"bbe8dcdb-21a5-45a2-8102-ab451b7172a7","Type":"ContainerStarted","Data":"d9feeb15ef61d1ed27afa665333fdefd4cbc93bb49216eaa1d23dabfda274e93"} Jul 30 00:46:23.758895 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:23.758861 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:23.759128 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:23.759097 2572 patch_prober.go:28] interesting pod/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 30 00:46:23.759172 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:23.759152 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" podUID="bbe8dcdb-21a5-45a2-8102-ab451b7172a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 30 00:46:24.759124 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:24.759069 2572 patch_prober.go:28] interesting pod/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 30 00:46:24.759711 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:24.759155 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" podUID="bbe8dcdb-21a5-45a2-8102-ab451b7172a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 30 00:46:25.759776 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:25.759737 2572 patch_prober.go:28] interesting pod/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 30 00:46:25.760225 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:25.759809 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" podUID="bbe8dcdb-21a5-45a2-8102-ab451b7172a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 30 00:46:26.758964 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:26.758926 2572 patch_prober.go:28] interesting pod/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 30 00:46:26.759142 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:26.758983 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" podUID="bbe8dcdb-21a5-45a2-8102-ab451b7172a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 30 00:46:27.763297 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:27.763263 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:27.763711 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:27.763637 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:27.764364 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:27.764343 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 00:46:33.030423 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.030393 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:46:33.461686 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.461652 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:46:33.621727 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.621696 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph"] Jul 30 00:46:33.622015 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.622001 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20ca2a62-7955-4174-9377-1d11d31728c9" containerName="istio-proxy" Jul 30 00:46:33.622059 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.622016 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="20ca2a62-7955-4174-9377-1d11d31728c9" containerName="istio-proxy" Jul 30 00:46:33.622059 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.622028 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20ca2a62-7955-4174-9377-1d11d31728c9" containerName="istio-proxy" Jul 30 00:46:33.622059 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.622034 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="20ca2a62-7955-4174-9377-1d11d31728c9" containerName="istio-proxy" Jul 30 00:46:33.622172 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.622082 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="20ca2a62-7955-4174-9377-1d11d31728c9" containerName="istio-proxy" Jul 30 00:46:33.622172 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.622089 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="20ca2a62-7955-4174-9377-1d11d31728c9" containerName="istio-proxy" Jul 30 00:46:33.623950 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.623931 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.627202 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.627176 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-0b2dae-openshift-default-dockercfg-672n6\"" Jul 30 00:46:33.638959 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.638938 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph"] Jul 30 00:46:33.652518 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.652476 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:46:33.741672 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.741570 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-istio-envoy\") pod \"e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.741672 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.741629 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7597295b-dca6-4d83-a443-c12cd95f581e-istio-token\") pod \"e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.741925 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.741690 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7597295b-dca6-4d83-a443-c12cd95f581e-istiod-ca-cert\") pod \"e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.741925 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.741726 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-workload-socket\") pod \"e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.741925 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.741760 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-credential-socket\") pod \"e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.741925 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.741825 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7597295b-dca6-4d83-a443-c12cd95f581e-istio-podinfo\") pod \"e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.741925 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.741893 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-istio-data\") pod \"e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.741925 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.741918 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmmqz\" (UniqueName: \"kubernetes.io/projected/7597295b-dca6-4d83-a443-c12cd95f581e-kube-api-access-qmmqz\") pod \"e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.742255 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.741942 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-workload-certs\") pod \"e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.843073 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.843034 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-istio-envoy\") pod \"e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.843073 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.843079 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7597295b-dca6-4d83-a443-c12cd95f581e-istio-token\") pod \"e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.843358 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.843277 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7597295b-dca6-4d83-a443-c12cd95f581e-istiod-ca-cert\") pod \"e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.843358 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.843313 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-workload-socket\") pod \"e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.843358 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.843332 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-credential-socket\") pod \"e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.843531 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.843374 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7597295b-dca6-4d83-a443-c12cd95f581e-istio-podinfo\") pod \"e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.843531 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.843445 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-istio-data\") pod \"e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.843531 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.843469 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmmqz\" (UniqueName: \"kubernetes.io/projected/7597295b-dca6-4d83-a443-c12cd95f581e-kube-api-access-qmmqz\") pod \"e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.843689 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.843662 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-workload-certs\") pod \"e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.843790 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.843695 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-workload-socket\") pod \"e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.843852 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.843782 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-credential-socket\") pod \"e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.843948 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.843927 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-istio-data\") pod \"e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.844013 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.843985 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-workload-certs\") pod \"e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.844053 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.844040 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7597295b-dca6-4d83-a443-c12cd95f581e-istiod-ca-cert\") pod \"e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.846146 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.846126 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-istio-envoy\") pod \"e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.846349 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.846329 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7597295b-dca6-4d83-a443-c12cd95f581e-istio-podinfo\") pod \"e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.851335 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.851311 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmmqz\" (UniqueName: \"kubernetes.io/projected/7597295b-dca6-4d83-a443-c12cd95f581e-kube-api-access-qmmqz\") pod \"e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.851448 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.851309 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7597295b-dca6-4d83-a443-c12cd95f581e-istio-token\") pod \"e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.934211 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.934183 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:33.953452 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:33.953417 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:46:34.067051 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:34.067019 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph"] Jul 30 00:46:34.068858 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:46:34.068833 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7597295b_dca6_4d83_a443_c12cd95f581e.slice/crio-38be4e973dbc847d17375ec3767a8dbb73f84ee60dbec524e8a739a2310c17d7 WatchSource:0}: Error finding container 38be4e973dbc847d17375ec3767a8dbb73f84ee60dbec524e8a739a2310c17d7: Status 404 returned error can't find the container with id 38be4e973dbc847d17375ec3767a8dbb73f84ee60dbec524e8a739a2310c17d7 Jul 30 00:46:34.071441 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:34.071410 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:46:34.071510 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:34.071483 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:46:34.071510 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:34.071507 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:46:34.285322 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:34.285225 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" event={"ID":"7597295b-dca6-4d83-a443-c12cd95f581e","Type":"ContainerStarted","Data":"de6d4b46a40113da7bdd78f8d8de006a1863a1bfe2463e0ce3163f04e7dbfc4a"} Jul 30 00:46:34.285322 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:34.285280 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" event={"ID":"7597295b-dca6-4d83-a443-c12cd95f581e","Type":"ContainerStarted","Data":"38be4e973dbc847d17375ec3767a8dbb73f84ee60dbec524e8a739a2310c17d7"} Jul 30 00:46:34.304180 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:34.304125 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" podStartSLOduration=1.304063929 podStartE2EDuration="1.304063929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:46:34.303186401 +0000 UTC m=+2681.806444657" watchObservedRunningTime="2026-07-30 00:46:34.304063929 +0000 UTC m=+2681.807322189" Jul 30 00:46:34.935072 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:34.935038 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:35.939506 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:35.939472 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:36.209050 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:36.208959 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:46:36.294397 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:36.294360 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:36.295185 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:36.295163 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:36.307805 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:36.307781 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:46:37.321880 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:37.321843 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:46:37.476728 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:37.476695 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:46:37.803037 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:37.803006 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:46:37.906411 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:37.906378 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:46:49.351499 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:49.351474 2572 generic.go:356] "Generic (PLEG): container finished" podID="7597295b-dca6-4d83-a443-c12cd95f581e" containerID="de6d4b46a40113da7bdd78f8d8de006a1863a1bfe2463e0ce3163f04e7dbfc4a" exitCode=0 Jul 30 00:46:49.351817 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:49.351542 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" event={"ID":"7597295b-dca6-4d83-a443-c12cd95f581e","Type":"ContainerDied","Data":"de6d4b46a40113da7bdd78f8d8de006a1863a1bfe2463e0ce3163f04e7dbfc4a"} Jul 30 00:46:49.351924 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:49.351910 2572 scope.go:117] "RemoveContainer" containerID="de6d4b46a40113da7bdd78f8d8de006a1863a1bfe2463e0ce3163f04e7dbfc4a" Jul 30 00:46:49.352871 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:49.352848 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:46:49.352935 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:49.352912 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:46:49.352976 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:49.352942 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:46:50.364219 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:50.364186 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" event={"ID":"7597295b-dca6-4d83-a443-c12cd95f581e","Type":"ContainerStarted","Data":"8f8f191e17ee833b3e650df11285f41b7b95a147f286e6ecbcfe12c7cfa33844"} Jul 30 00:46:50.934984 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:50.934899 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:50.936262 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:50.936228 2572 patch_prober.go:28] interesting pod/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 30 00:46:50.936408 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:50.936288 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" podUID="7597295b-dca6-4d83-a443-c12cd95f581e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 30 00:46:51.294956 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:51.294871 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:51.935251 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:51.935211 2572 patch_prober.go:28] interesting pod/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 30 00:46:51.935775 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:51.935286 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" podUID="7597295b-dca6-4d83-a443-c12cd95f581e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 30 00:46:52.944581 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:52.944538 2572 patch_prober.go:28] interesting pod/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 30 00:46:52.945131 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:52.944630 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" podUID="7597295b-dca6-4d83-a443-c12cd95f581e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 30 00:46:53.935172 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:53.935133 2572 patch_prober.go:28] interesting pod/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 30 00:46:53.935458 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:53.935428 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" podUID="7597295b-dca6-4d83-a443-c12cd95f581e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 30 00:46:54.938754 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:54.938725 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:46:54.939738 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:46:54.939718 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 00:47:01.240828 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:47:01.240786 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:47:10.225529 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:47:10.225487 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:47:50.600375 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:47:50.600335 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:47:52.121722 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:47:52.121684 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:47:59.239738 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:47:59.239699 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:49:35.240047 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:49:35.240013 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:49:35.293230 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:49:35.293201 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:49:53.876129 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:49:53.876030 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:50:20.994089 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:50:20.994048 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:51:19.754225 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:51:19.754181 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:52:06.386224 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:52:06.386181 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:52:07.223160 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:52:07.223130 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:52:13.284719 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:52:13.284687 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:52:40.835303 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:52:40.835271 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:53:46.018416 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:53:46.018386 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:53:48.607326 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:53:48.607296 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:54:07.234266 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:54:07.234235 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:54:32.148041 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:54:32.148007 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:54:36.429328 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:54:36.429291 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:55:58.732648 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:55:58.732565 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:56:00.405631 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:00.405596 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:56:18.725197 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:18.725161 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:56:19.017969 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:19.017895 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:56:19.575727 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:19.575696 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:56:21.993839 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:21.993811 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:56:22.084010 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:22.083982 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:56:23.071572 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:23.071541 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj"] Jul 30 00:56:23.074693 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:23.074674 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj" Jul 30 00:56:23.077252 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:23.077227 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-7fa598-serving-cert\"" Jul 30 00:56:23.084968 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:23.084945 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj"] Jul 30 00:56:23.104268 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:23.104244 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:56:23.164100 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:23.164078 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj\" (UID: \"c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj" Jul 30 00:56:23.164232 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:23.164148 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj\" (UID: \"c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj" Jul 30 00:56:23.164232 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:23.164197 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8g47\" (UniqueName: \"kubernetes.io/projected/c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4-kube-api-access-k8g47\") pod \"maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj\" (UID: \"c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj" Jul 30 00:56:23.264966 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:23.264930 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj\" (UID: \"c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj" Jul 30 00:56:23.265211 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:23.264997 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8g47\" (UniqueName: \"kubernetes.io/projected/c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4-kube-api-access-k8g47\") pod \"maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj\" (UID: \"c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj" Jul 30 00:56:23.265211 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:23.265043 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj\" (UID: \"c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj" Jul 30 00:56:23.265211 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:56:23.265097 2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-serving-cert: secret "maas-api-e2e-ipp-a-7fa598-serving-cert" not found Jul 30 00:56:23.265211 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:56:23.265198 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4-maas-api-tls podName:c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4 nodeName:}" failed. No retries permitted until 2026-07-30 00:56:23.765176164 +0000 UTC m=+3271.268434416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4-maas-api-tls") pod "maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj" (UID: "c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4") : secret "maas-api-e2e-ipp-a-7fa598-serving-cert" not found Jul 30 00:56:23.265701 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:23.265683 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj\" (UID: \"c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj" Jul 30 00:56:23.273275 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:23.273252 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8g47\" (UniqueName: \"kubernetes.io/projected/c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4-kube-api-access-k8g47\") pod \"maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj\" (UID: \"c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj" Jul 30 00:56:23.638644 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:23.638608 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:56:23.739970 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:23.739935 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:56:23.769171 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:23.769144 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj\" (UID: \"c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj" Jul 30 00:56:23.771709 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:23.771683 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj\" (UID: \"c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj" Jul 30 00:56:23.986739 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:23.986657 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj" Jul 30 00:56:24.114192 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:24.114135 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj"] Jul 30 00:56:24.116474 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:56:24.116443 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6b6080a_9a6e_4513_aa45_6c7ea6b8d6a4.slice/crio-c6ebdcbec774a82306c5afdf7121b8ae138cbaa5c144b46c41648cfa4275c634 WatchSource:0}: Error finding container c6ebdcbec774a82306c5afdf7121b8ae138cbaa5c144b46c41648cfa4275c634: Status 404 returned error can't find the container with id c6ebdcbec774a82306c5afdf7121b8ae138cbaa5c144b46c41648cfa4275c634 Jul 30 00:56:24.117865 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:24.117848 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 00:56:24.702344 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:24.702313 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj" event={"ID":"c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4","Type":"ContainerStarted","Data":"c33ab2cef96a0162707f19a57c128d5c6df6532f15603c3052509811668362b7"} Jul 30 00:56:24.702496 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:24.702364 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj" event={"ID":"c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4","Type":"ContainerStarted","Data":"c6ebdcbec774a82306c5afdf7121b8ae138cbaa5c144b46c41648cfa4275c634"} Jul 30 00:56:24.702496 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:24.702414 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj" Jul 30 00:56:24.718400 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:24.718357 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj" podStartSLOduration=1.273674221 podStartE2EDuration="1.718343032s" totalImagesPullingTime="444.668811ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:56:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:56:24.717157888 +0000 UTC m=+3272.220416149" watchObservedRunningTime="2026-07-30 00:56:24.718343032 +0000 UTC m=+3272.221601291" Jul 30 00:56:30.711949 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:30.711919 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj" Jul 30 00:56:31.371858 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:31.371820 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:56:52.086297 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:52.086266 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:56:52.758815 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:52.758783 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:56:53.072873 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:53.072776 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:56:53.321419 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:53.321381 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:56:56.209078 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:56.209041 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:56:56.309494 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:56.309460 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:56:57.296620 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:57.296571 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc"] Jul 30 00:56:57.300461 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:57.300441 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc" Jul 30 00:56:57.303163 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:57.303139 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-edaa93-serving-cert\"" Jul 30 00:56:57.308670 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:57.308650 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc"] Jul 30 00:56:57.325141 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:57.325103 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:56:57.340320 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:57.340296 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e445d9-72a1-45b0-aa61-449f8c6f66f5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc\" (UID: \"33e445d9-72a1-45b0-aa61-449f8c6f66f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc" Jul 30 00:56:57.340457 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:57.340337 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e445d9-72a1-45b0-aa61-449f8c6f66f5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc\" (UID: \"33e445d9-72a1-45b0-aa61-449f8c6f66f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc" Jul 30 00:56:57.340528 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:57.340460 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzlht\" (UniqueName: \"kubernetes.io/projected/33e445d9-72a1-45b0-aa61-449f8c6f66f5-kube-api-access-xzlht\") pod \"maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc\" (UID: \"33e445d9-72a1-45b0-aa61-449f8c6f66f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc" Jul 30 00:56:57.431890 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:57.431858 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:56:57.441073 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:57.441045 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzlht\" (UniqueName: \"kubernetes.io/projected/33e445d9-72a1-45b0-aa61-449f8c6f66f5-kube-api-access-xzlht\") pod \"maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc\" (UID: \"33e445d9-72a1-45b0-aa61-449f8c6f66f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc" Jul 30 00:56:57.441216 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:57.441096 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e445d9-72a1-45b0-aa61-449f8c6f66f5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc\" (UID: \"33e445d9-72a1-45b0-aa61-449f8c6f66f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc" Jul 30 00:56:57.441216 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:57.441150 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e445d9-72a1-45b0-aa61-449f8c6f66f5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc\" (UID: \"33e445d9-72a1-45b0-aa61-449f8c6f66f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc" Jul 30 00:56:57.441857 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:57.441836 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e445d9-72a1-45b0-aa61-449f8c6f66f5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc\" (UID: \"33e445d9-72a1-45b0-aa61-449f8c6f66f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc" Jul 30 00:56:57.443512 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:57.443494 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e445d9-72a1-45b0-aa61-449f8c6f66f5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc\" (UID: \"33e445d9-72a1-45b0-aa61-449f8c6f66f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc" Jul 30 00:56:57.448771 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:57.448750 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzlht\" (UniqueName: \"kubernetes.io/projected/33e445d9-72a1-45b0-aa61-449f8c6f66f5-kube-api-access-xzlht\") pod \"maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc\" (UID: \"33e445d9-72a1-45b0-aa61-449f8c6f66f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc" Jul 30 00:56:57.613017 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:57.612987 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc" Jul 30 00:56:57.742831 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:57.742775 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc"] Jul 30 00:56:57.744871 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:56:57.744845 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e445d9_72a1_45b0_aa61_449f8c6f66f5.slice/crio-7cd0e32400dc6b7f7636f3cfb2d12230890f34c8405d6cf92758d8b146ffd362 WatchSource:0}: Error finding container 7cd0e32400dc6b7f7636f3cfb2d12230890f34c8405d6cf92758d8b146ffd362: Status 404 returned error can't find the container with id 7cd0e32400dc6b7f7636f3cfb2d12230890f34c8405d6cf92758d8b146ffd362 Jul 30 00:56:57.823072 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:57.823029 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:56:57.842036 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:57.842001 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc" event={"ID":"33e445d9-72a1-45b0-aa61-449f8c6f66f5","Type":"ContainerStarted","Data":"7cd0e32400dc6b7f7636f3cfb2d12230890f34c8405d6cf92758d8b146ffd362"} Jul 30 00:56:57.919738 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:57.919662 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:56:58.853430 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:58.853392 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc" event={"ID":"33e445d9-72a1-45b0-aa61-449f8c6f66f5","Type":"ContainerStarted","Data":"cd2f673e27d722a9c62ec6e6381070d84d6322c70a45c81e78c68106df7d5ed5"} Jul 30 00:56:58.853852 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:58.853507 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc" Jul 30 00:56:58.869373 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:56:58.869328 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc" podStartSLOduration=1.489489585 podStartE2EDuration="1.869314862s" totalImagesPullingTime="379.825277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:56:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:56:58.868451807 +0000 UTC m=+3306.371710070" watchObservedRunningTime="2026-07-30 00:56:58.869314862 +0000 UTC m=+3306.372573123" Jul 30 00:57:04.862806 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:57:04.862779 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc" Jul 30 00:57:21.183354 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:57:21.183321 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:57:48.001012 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:57:48.000978 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:58:28.925867 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:28.925831 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:58:29.192619 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.192538 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4"] Jul 30 00:58:29.195895 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.195876 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.198512 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.198490 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-859003-openshift-default-dockercfg-46wlz\"" Jul 30 00:58:29.205685 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.205658 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4"] Jul 30 00:58:29.215815 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.215794 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:58:29.243290 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.243258 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-workload-certs\") pod \"e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.243290 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.243296 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istio-data\") pod \"e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.243514 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.243328 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vzv8\" (UniqueName: \"kubernetes.io/projected/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-kube-api-access-4vzv8\") pod \"e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.243514 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.243401 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-credential-socket\") pod \"e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.243514 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.243428 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istio-envoy\") pod \"e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.243514 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.243445 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istio-podinfo\") pod \"e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.243514 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.243461 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istiod-ca-cert\") pod \"e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.243514 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.243510 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istio-token\") pod \"e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.243752 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.243542 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-workload-socket\") pod \"e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.344788 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.344752 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-workload-socket\") pod \"e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.344996 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.344827 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-workload-certs\") pod \"e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.344996 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.344845 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istio-data\") pod \"e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.344996 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.344870 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vzv8\" (UniqueName: \"kubernetes.io/projected/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-kube-api-access-4vzv8\") pod \"e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.344996 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.344902 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-credential-socket\") pod \"e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.344996 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.344917 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istio-envoy\") pod \"e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.344996 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.344933 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istio-podinfo\") pod \"e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.345372 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.345051 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istiod-ca-cert\") pod \"e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.345372 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.345095 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istio-token\") pod \"e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.345372 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.345265 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istio-data\") pod \"e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.345372 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.345321 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-workload-socket\") pod \"e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.345534 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.345376 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-workload-certs\") pod \"e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.345665 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.345633 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-credential-socket\") pod \"e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.345882 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.345863 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istiod-ca-cert\") pod \"e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.347174 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.347154 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istio-envoy\") pod \"e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.347421 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.347402 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istio-podinfo\") pod \"e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.352557 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.352534 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istio-token\") pod \"e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.353154 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.353137 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vzv8\" (UniqueName: \"kubernetes.io/projected/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-kube-api-access-4vzv8\") pod \"e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.508028 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.507934 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:29.638522 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.638487 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4"] Jul 30 00:58:29.642736 ip-10-0-135-118 kubenswrapper[2572]: W0730 00:58:29.642705 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fdc83fa_b7d8_4c85_a844_7a0aa2494abe.slice/crio-c4034544881c0604aa51ff1a7a813cb3022adc57d0153edd89a404472df535a7 WatchSource:0}: Error finding container c4034544881c0604aa51ff1a7a813cb3022adc57d0153edd89a404472df535a7: Status 404 returned error can't find the container with id c4034544881c0604aa51ff1a7a813cb3022adc57d0153edd89a404472df535a7 Jul 30 00:58:29.645004 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.644973 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:58:29.645082 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.645037 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:58:29.645082 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.645062 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:58:29.767271 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:29.767179 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:58:30.243855 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:30.243811 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" event={"ID":"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe","Type":"ContainerStarted","Data":"53c9c1a265da0c73a1f3f2a1ce01bd30561de8f5dd82e99b35df3963ffd0be51"} Jul 30 00:58:30.244337 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:30.243864 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" event={"ID":"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe","Type":"ContainerStarted","Data":"c4034544881c0604aa51ff1a7a813cb3022adc57d0153edd89a404472df535a7"} Jul 30 00:58:30.264259 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:30.264217 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" podStartSLOduration=1.264201553 podStartE2EDuration="1.264201553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 00:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 00:58:30.262594133 +0000 UTC m=+3397.765852393" watchObservedRunningTime="2026-07-30 00:58:30.264201553 +0000 UTC m=+3397.767459816" Jul 30 00:58:30.508999 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:30.508920 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:30.513544 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:30.513523 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:31.247709 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:31.247679 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:31.248632 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:31.248603 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:32.173204 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:32.173151 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:58:32.236965 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:32.236936 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:58:33.342768 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:33.342734 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:58:33.567007 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:33.566972 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:58:33.862505 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:33.862473 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:58:33.963162 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:33.963130 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:58:45.314976 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:45.314871 2572 generic.go:356] "Generic (PLEG): container finished" podID="8fdc83fa-b7d8-4c85-a844-7a0aa2494abe" containerID="53c9c1a265da0c73a1f3f2a1ce01bd30561de8f5dd82e99b35df3963ffd0be51" exitCode=0 Jul 30 00:58:45.314976 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:45.314950 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" event={"ID":"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe","Type":"ContainerDied","Data":"53c9c1a265da0c73a1f3f2a1ce01bd30561de8f5dd82e99b35df3963ffd0be51"} Jul 30 00:58:45.315563 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:45.315487 2572 scope.go:117] "RemoveContainer" containerID="53c9c1a265da0c73a1f3f2a1ce01bd30561de8f5dd82e99b35df3963ffd0be51" Jul 30 00:58:45.316612 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:45.316580 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:58:45.316694 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:45.316663 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:58:45.316731 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:45.316695 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 30 00:58:46.248431 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:46.248381 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:46.321361 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:46.321332 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" event={"ID":"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe","Type":"ContainerStarted","Data":"d73e3e223ddbd93550935e3a5cf7589ce09543b3e2934bc8e0b111c23655cb6c"} Jul 30 00:58:46.508959 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:46.508876 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:46.509796 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:46.509770 2572 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 30 00:58:46.509916 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:46.509822 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" podUID="8fdc83fa-b7d8-4c85-a844-7a0aa2494abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 30 00:58:47.509227 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:47.509192 2572 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 30 00:58:47.509624 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:47.509244 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" podUID="8fdc83fa-b7d8-4c85-a844-7a0aa2494abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 30 00:58:48.509222 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:48.509183 2572 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 30 00:58:48.509409 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:48.509245 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" podUID="8fdc83fa-b7d8-4c85-a844-7a0aa2494abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 30 00:58:49.508756 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:49.508711 2572 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 30 00:58:49.508941 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:49.508800 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" podUID="8fdc83fa-b7d8-4c85-a844-7a0aa2494abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 30 00:58:50.512639 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:50.512603 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:50.513077 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:50.512913 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:50.513533 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:50.513511 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:58:57.155700 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:58:57.155666 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:59:03.907557 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:03.907418 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:59:03.998990 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:03.998955 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:59:04.061607 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:04.061574 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:59:04.419509 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:04.419473 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:59:10.065153 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:10.065095 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:59:10.165537 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:10.165507 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:59:10.305283 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:10.305249 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4"] Jul 30 00:59:10.305554 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:10.305532 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" podUID="8fdc83fa-b7d8-4c85-a844-7a0aa2494abe" containerName="istio-proxy" containerID="cri-o://d73e3e223ddbd93550935e3a5cf7589ce09543b3e2934bc8e0b111c23655cb6c" gracePeriod=30 Jul 30 00:59:10.324679 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:10.324587 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:59:10.534353 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:10.534329 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:59:15.452785 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.452752 2572 generic.go:356] "Generic (PLEG): container finished" podID="8fdc83fa-b7d8-4c85-a844-7a0aa2494abe" containerID="d73e3e223ddbd93550935e3a5cf7589ce09543b3e2934bc8e0b111c23655cb6c" exitCode=0 Jul 30 00:59:15.453171 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.452826 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" event={"ID":"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe","Type":"ContainerDied","Data":"d73e3e223ddbd93550935e3a5cf7589ce09543b3e2934bc8e0b111c23655cb6c"} Jul 30 00:59:15.453171 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.452920 2572 scope.go:117] "RemoveContainer" containerID="53c9c1a265da0c73a1f3f2a1ce01bd30561de8f5dd82e99b35df3963ffd0be51" Jul 30 00:59:15.547853 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.547831 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:59:15.590398 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.590372 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istio-envoy\") pod \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " Jul 30 00:59:15.590572 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.590427 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-workload-certs\") pod \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " Jul 30 00:59:15.590572 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.590457 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vzv8\" (UniqueName: \"kubernetes.io/projected/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-kube-api-access-4vzv8\") pod \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " Jul 30 00:59:15.590572 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.590474 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istio-podinfo\") pod \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " Jul 30 00:59:15.590572 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.590489 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-workload-socket\") pod \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " Jul 30 00:59:15.590572 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.590514 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istiod-ca-cert\") pod \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " Jul 30 00:59:15.590572 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.590532 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istio-data\") pod \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " Jul 30 00:59:15.590572 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.590552 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istio-token\") pod \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " Jul 30 00:59:15.590963 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.590609 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-credential-socket\") pod \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\" (UID: \"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe\") " Jul 30 00:59:15.591042 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.591005 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8fdc83fa-b7d8-4c85-a844-7a0aa2494abe" (UID: "8fdc83fa-b7d8-4c85-a844-7a0aa2494abe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:59:15.591120 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.591042 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8fdc83fa-b7d8-4c85-a844-7a0aa2494abe" (UID: "8fdc83fa-b7d8-4c85-a844-7a0aa2494abe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:59:15.591282 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.591259 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8fdc83fa-b7d8-4c85-a844-7a0aa2494abe" (UID: "8fdc83fa-b7d8-4c85-a844-7a0aa2494abe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:59:15.591348 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.591316 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8fdc83fa-b7d8-4c85-a844-7a0aa2494abe" (UID: "8fdc83fa-b7d8-4c85-a844-7a0aa2494abe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:59:15.591616 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.591589 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istio-data" (OuterVolumeSpecName: "istio-data") pod "8fdc83fa-b7d8-4c85-a844-7a0aa2494abe" (UID: "8fdc83fa-b7d8-4c85-a844-7a0aa2494abe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:59:15.593056 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.593037 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8fdc83fa-b7d8-4c85-a844-7a0aa2494abe" (UID: "8fdc83fa-b7d8-4c85-a844-7a0aa2494abe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 00:59:15.593196 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.593039 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8fdc83fa-b7d8-4c85-a844-7a0aa2494abe" (UID: "8fdc83fa-b7d8-4c85-a844-7a0aa2494abe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 00:59:15.593196 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.593056 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-kube-api-access-4vzv8" (OuterVolumeSpecName: "kube-api-access-4vzv8") pod "8fdc83fa-b7d8-4c85-a844-7a0aa2494abe" (UID: "8fdc83fa-b7d8-4c85-a844-7a0aa2494abe"). InnerVolumeSpecName "kube-api-access-4vzv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:59:15.593396 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.593375 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istio-token" (OuterVolumeSpecName: "istio-token") pod "8fdc83fa-b7d8-4c85-a844-7a0aa2494abe" (UID: "8fdc83fa-b7d8-4c85-a844-7a0aa2494abe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:59:15.691578 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.691547 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-credential-socket\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:59:15.691578 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.691576 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istio-envoy\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:59:15.691578 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.691585 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-workload-certs\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:59:15.691798 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.691593 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vzv8\" (UniqueName: \"kubernetes.io/projected/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-kube-api-access-4vzv8\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:59:15.691798 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.691604 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istio-podinfo\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:59:15.691798 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.691613 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-workload-socket\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:59:15.691798 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.691621 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istiod-ca-cert\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:59:15.691798 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.691629 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istio-data\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:59:15.691798 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:15.691636 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe-istio-token\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:59:16.458081 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:16.458058 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" Jul 30 00:59:16.458494 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:16.458050 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4" event={"ID":"8fdc83fa-b7d8-4c85-a844-7a0aa2494abe","Type":"ContainerDied","Data":"c4034544881c0604aa51ff1a7a813cb3022adc57d0153edd89a404472df535a7"} Jul 30 00:59:16.458494 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:16.458170 2572 scope.go:117] "RemoveContainer" containerID="d73e3e223ddbd93550935e3a5cf7589ce09543b3e2934bc8e0b111c23655cb6c" Jul 30 00:59:16.481756 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:16.481731 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4"] Jul 30 00:59:16.485376 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:16.485352 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-859003-openshift-default-7d9b744d98-dzsg4"] Jul 30 00:59:16.992356 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:16.992325 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fdc83fa-b7d8-4c85-a844-7a0aa2494abe" path="/var/lib/kubelet/pods/8fdc83fa-b7d8-4c85-a844-7a0aa2494abe/volumes" Jul 30 00:59:29.198342 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:29.198305 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj"] Jul 30 00:59:29.198819 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:29.198527 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj" podUID="c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4" containerName="maas-api" containerID="cri-o://c33ab2cef96a0162707f19a57c128d5c6df6532f15603c3052509811668362b7" gracePeriod=30 Jul 30 00:59:29.232486 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:29.232460 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:59:29.302525 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:29.302493 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:59:29.365392 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:29.365354 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:59:29.423274 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:29.423242 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:59:29.450509 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:29.450420 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj" Jul 30 00:59:29.515133 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:29.515080 2572 generic.go:356] "Generic (PLEG): container finished" podID="c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4" containerID="c33ab2cef96a0162707f19a57c128d5c6df6532f15603c3052509811668362b7" exitCode=0 Jul 30 00:59:29.515133 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:29.515136 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj" Jul 30 00:59:29.515346 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:29.515167 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj" event={"ID":"c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4","Type":"ContainerDied","Data":"c33ab2cef96a0162707f19a57c128d5c6df6532f15603c3052509811668362b7"} Jul 30 00:59:29.515346 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:29.515226 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj" event={"ID":"c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4","Type":"ContainerDied","Data":"c6ebdcbec774a82306c5afdf7121b8ae138cbaa5c144b46c41648cfa4275c634"} Jul 30 00:59:29.515346 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:29.515257 2572 scope.go:117] "RemoveContainer" containerID="c33ab2cef96a0162707f19a57c128d5c6df6532f15603c3052509811668362b7" Jul 30 00:59:29.523045 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:29.523023 2572 scope.go:117] "RemoveContainer" containerID="c33ab2cef96a0162707f19a57c128d5c6df6532f15603c3052509811668362b7" Jul 30 00:59:29.523320 ip-10-0-135-118 kubenswrapper[2572]: E0730 00:59:29.523300 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c33ab2cef96a0162707f19a57c128d5c6df6532f15603c3052509811668362b7\": container with ID starting with c33ab2cef96a0162707f19a57c128d5c6df6532f15603c3052509811668362b7 not found: ID does not exist" containerID="c33ab2cef96a0162707f19a57c128d5c6df6532f15603c3052509811668362b7" Jul 30 00:59:29.523395 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:29.523329 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c33ab2cef96a0162707f19a57c128d5c6df6532f15603c3052509811668362b7"} err="failed to get container status \"c33ab2cef96a0162707f19a57c128d5c6df6532f15603c3052509811668362b7\": rpc error: code = NotFound desc = could not find container \"c33ab2cef96a0162707f19a57c128d5c6df6532f15603c3052509811668362b7\": container with ID starting with c33ab2cef96a0162707f19a57c128d5c6df6532f15603c3052509811668362b7 not found: ID does not exist" Jul 30 00:59:29.617987 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:29.617955 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8g47\" (UniqueName: \"kubernetes.io/projected/c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4-kube-api-access-k8g47\") pod \"c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4\" (UID: \"c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4\") " Jul 30 00:59:29.618176 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:29.618022 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4-maas-api-tls\") pod \"c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4\" (UID: \"c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4\") " Jul 30 00:59:29.618176 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:29.618083 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4-openshift-service-ca\") pod \"c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4\" (UID: \"c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4\") " Jul 30 00:59:29.618503 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:29.618479 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4" (UID: "c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 00:59:29.620051 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:29.620029 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4" (UID: "c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 00:59:29.620153 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:29.620065 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4-kube-api-access-k8g47" (OuterVolumeSpecName: "kube-api-access-k8g47") pod "c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4" (UID: "c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4"). InnerVolumeSpecName "kube-api-access-k8g47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 00:59:29.719219 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:29.719140 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4-maas-api-tls\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:59:29.719219 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:29.719167 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4-openshift-service-ca\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:59:29.719219 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:29.719178 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8g47\" (UniqueName: \"kubernetes.io/projected/c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4-kube-api-access-k8g47\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 00:59:29.767552 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:29.767519 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:59:29.836247 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:29.836216 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj"] Jul 30 00:59:29.840888 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:29.840864 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7fa598-677f8f8979-rrstj"] Jul 30 00:59:30.992267 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:30.992241 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4" path="/var/lib/kubelet/pods/c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4/volumes" Jul 30 00:59:34.847716 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:34.847677 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 00:59:59.640905 ip-10-0-135-118 kubenswrapper[2572]: I0730 00:59:59.640869 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:00:11.657119 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:11.657058 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:00:12.135279 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:12.135239 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:00:17.813243 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:17.813217 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:00:31.713512 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:31.713466 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:00:31.773400 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:31.773344 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:00:31.980913 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:31.980824 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:00:32.221722 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:32.221691 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:00:37.025709 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:37.025676 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc"] Jul 30 01:00:37.026355 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:37.025899 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc" podUID="33e445d9-72a1-45b0-aa61-449f8c6f66f5" containerName="maas-api" containerID="cri-o://cd2f673e27d722a9c62ec6e6381070d84d6322c70a45c81e78c68106df7d5ed5" gracePeriod=30 Jul 30 01:00:37.051185 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:37.051158 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:00:37.131700 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:37.131670 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:00:37.193314 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:37.193283 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:00:37.250132 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:37.250081 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:00:37.286527 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:37.286460 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc" Jul 30 01:00:37.299217 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:37.299192 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzlht\" (UniqueName: \"kubernetes.io/projected/33e445d9-72a1-45b0-aa61-449f8c6f66f5-kube-api-access-xzlht\") pod \"33e445d9-72a1-45b0-aa61-449f8c6f66f5\" (UID: \"33e445d9-72a1-45b0-aa61-449f8c6f66f5\") " Jul 30 01:00:37.299353 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:37.299239 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e445d9-72a1-45b0-aa61-449f8c6f66f5-openshift-service-ca\") pod \"33e445d9-72a1-45b0-aa61-449f8c6f66f5\" (UID: \"33e445d9-72a1-45b0-aa61-449f8c6f66f5\") " Jul 30 01:00:37.299353 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:37.299282 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e445d9-72a1-45b0-aa61-449f8c6f66f5-maas-api-tls\") pod \"33e445d9-72a1-45b0-aa61-449f8c6f66f5\" (UID: \"33e445d9-72a1-45b0-aa61-449f8c6f66f5\") " Jul 30 01:00:37.299572 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:37.299550 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e445d9-72a1-45b0-aa61-449f8c6f66f5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "33e445d9-72a1-45b0-aa61-449f8c6f66f5" (UID: "33e445d9-72a1-45b0-aa61-449f8c6f66f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 01:00:37.301381 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:37.301349 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e445d9-72a1-45b0-aa61-449f8c6f66f5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "33e445d9-72a1-45b0-aa61-449f8c6f66f5" (UID: "33e445d9-72a1-45b0-aa61-449f8c6f66f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 01:00:37.301564 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:37.301476 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e445d9-72a1-45b0-aa61-449f8c6f66f5-kube-api-access-xzlht" (OuterVolumeSpecName: "kube-api-access-xzlht") pod "33e445d9-72a1-45b0-aa61-449f8c6f66f5" (UID: "33e445d9-72a1-45b0-aa61-449f8c6f66f5"). InnerVolumeSpecName "kube-api-access-xzlht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 01:00:37.399983 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:37.399951 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e445d9-72a1-45b0-aa61-449f8c6f66f5-maas-api-tls\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 01:00:37.399983 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:37.399979 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzlht\" (UniqueName: \"kubernetes.io/projected/33e445d9-72a1-45b0-aa61-449f8c6f66f5-kube-api-access-xzlht\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 01:00:37.399983 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:37.399989 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e445d9-72a1-45b0-aa61-449f8c6f66f5-openshift-service-ca\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 01:00:37.800525 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:37.800493 2572 generic.go:356] "Generic (PLEG): container finished" podID="33e445d9-72a1-45b0-aa61-449f8c6f66f5" containerID="cd2f673e27d722a9c62ec6e6381070d84d6322c70a45c81e78c68106df7d5ed5" exitCode=0 Jul 30 01:00:37.800702 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:37.800539 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc" Jul 30 01:00:37.800702 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:37.800582 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc" event={"ID":"33e445d9-72a1-45b0-aa61-449f8c6f66f5","Type":"ContainerDied","Data":"cd2f673e27d722a9c62ec6e6381070d84d6322c70a45c81e78c68106df7d5ed5"} Jul 30 01:00:37.800702 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:37.800645 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc" event={"ID":"33e445d9-72a1-45b0-aa61-449f8c6f66f5","Type":"ContainerDied","Data":"7cd0e32400dc6b7f7636f3cfb2d12230890f34c8405d6cf92758d8b146ffd362"} Jul 30 01:00:37.800702 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:37.800671 2572 scope.go:117] "RemoveContainer" containerID="cd2f673e27d722a9c62ec6e6381070d84d6322c70a45c81e78c68106df7d5ed5" Jul 30 01:00:37.809962 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:37.809942 2572 scope.go:117] "RemoveContainer" containerID="cd2f673e27d722a9c62ec6e6381070d84d6322c70a45c81e78c68106df7d5ed5" Jul 30 01:00:37.810303 ip-10-0-135-118 kubenswrapper[2572]: E0730 01:00:37.810260 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd2f673e27d722a9c62ec6e6381070d84d6322c70a45c81e78c68106df7d5ed5\": container with ID starting with cd2f673e27d722a9c62ec6e6381070d84d6322c70a45c81e78c68106df7d5ed5 not found: ID does not exist" containerID="cd2f673e27d722a9c62ec6e6381070d84d6322c70a45c81e78c68106df7d5ed5" Jul 30 01:00:37.810538 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:37.810299 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd2f673e27d722a9c62ec6e6381070d84d6322c70a45c81e78c68106df7d5ed5"} err="failed to get container status \"cd2f673e27d722a9c62ec6e6381070d84d6322c70a45c81e78c68106df7d5ed5\": rpc error: code = NotFound desc = could not find container \"cd2f673e27d722a9c62ec6e6381070d84d6322c70a45c81e78c68106df7d5ed5\": container with ID starting with cd2f673e27d722a9c62ec6e6381070d84d6322c70a45c81e78c68106df7d5ed5 not found: ID does not exist" Jul 30 01:00:37.826760 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:37.826732 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc"] Jul 30 01:00:37.828465 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:37.828443 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-edaa93-557745bdb5-55lrc"] Jul 30 01:00:38.993159 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:38.993127 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33e445d9-72a1-45b0-aa61-449f8c6f66f5" path="/var/lib/kubelet/pods/33e445d9-72a1-45b0-aa61-449f8c6f66f5/volumes" Jul 30 01:00:39.693205 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:39.693175 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:00:43.539405 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:43.539368 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:00:43.639834 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:43.639758 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:00:43.744283 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:43.744255 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:00:43.965692 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:43.965651 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:00:54.617438 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:54.617404 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:00:54.680340 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:54.680308 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:00:54.741017 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:54.740981 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:00:54.843619 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:54.843583 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:00:55.173897 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:00:55.173866 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:01:01.010295 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:01.010232 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:01:01.106159 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:01.106125 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:01:01.243832 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:01.243796 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv"] Jul 30 01:01:01.244088 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:01.244055 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" podUID="bbe8dcdb-21a5-45a2-8102-ab451b7172a7" containerName="istio-proxy" containerID="cri-o://d9feeb15ef61d1ed27afa665333fdefd4cbc93bb49216eaa1d23dabfda274e93" gracePeriod=30 Jul 30 01:01:01.263600 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:01.263498 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:01:01.486965 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:01.486930 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:01:06.491682 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.491654 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 01:01:06.655098 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.655060 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istio-token\") pod \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " Jul 30 01:01:06.655098 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.655101 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-credential-socket\") pod \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " Jul 30 01:01:06.655398 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.655170 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istio-podinfo\") pod \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " Jul 30 01:01:06.655398 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.655191 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4l8z\" (UniqueName: \"kubernetes.io/projected/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-kube-api-access-l4l8z\") pod \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " Jul 30 01:01:06.655398 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.655231 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-workload-certs\") pod \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " Jul 30 01:01:06.655398 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.655370 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istiod-ca-cert\") pod \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " Jul 30 01:01:06.655628 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.655414 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-workload-socket\") pod \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " Jul 30 01:01:06.655628 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.655466 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istio-envoy\") pod \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " Jul 30 01:01:06.655628 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.655486 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bbe8dcdb-21a5-45a2-8102-ab451b7172a7" (UID: "bbe8dcdb-21a5-45a2-8102-ab451b7172a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:01:06.655628 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.655505 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bbe8dcdb-21a5-45a2-8102-ab451b7172a7" (UID: "bbe8dcdb-21a5-45a2-8102-ab451b7172a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:01:06.655628 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.655515 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istio-data\") pod \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\" (UID: \"bbe8dcdb-21a5-45a2-8102-ab451b7172a7\") " Jul 30 01:01:06.655878 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.655705 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bbe8dcdb-21a5-45a2-8102-ab451b7172a7" (UID: "bbe8dcdb-21a5-45a2-8102-ab451b7172a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:01:06.655878 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.655741 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bbe8dcdb-21a5-45a2-8102-ab451b7172a7" (UID: "bbe8dcdb-21a5-45a2-8102-ab451b7172a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 01:01:06.655990 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.655941 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-credential-socket\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 01:01:06.655990 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.655965 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-workload-certs\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 01:01:06.655990 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.655981 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istiod-ca-cert\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 01:01:06.656171 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.655995 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-workload-socket\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 01:01:06.656479 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.656456 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istio-data" (OuterVolumeSpecName: "istio-data") pod "bbe8dcdb-21a5-45a2-8102-ab451b7172a7" (UID: "bbe8dcdb-21a5-45a2-8102-ab451b7172a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:01:06.657411 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.657378 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istio-token" (OuterVolumeSpecName: "istio-token") pod "bbe8dcdb-21a5-45a2-8102-ab451b7172a7" (UID: "bbe8dcdb-21a5-45a2-8102-ab451b7172a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 01:01:06.657495 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.657461 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bbe8dcdb-21a5-45a2-8102-ab451b7172a7" (UID: "bbe8dcdb-21a5-45a2-8102-ab451b7172a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 01:01:06.657764 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.657739 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-kube-api-access-l4l8z" (OuterVolumeSpecName: "kube-api-access-l4l8z") pod "bbe8dcdb-21a5-45a2-8102-ab451b7172a7" (UID: "bbe8dcdb-21a5-45a2-8102-ab451b7172a7"). InnerVolumeSpecName "kube-api-access-l4l8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 01:01:06.657764 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.657744 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bbe8dcdb-21a5-45a2-8102-ab451b7172a7" (UID: "bbe8dcdb-21a5-45a2-8102-ab451b7172a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:01:06.756631 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.756594 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istio-envoy\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 01:01:06.756631 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.756621 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istio-data\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 01:01:06.756631 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.756636 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istio-token\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 01:01:06.756864 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.756644 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-istio-podinfo\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 01:01:06.756864 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.756652 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4l8z\" (UniqueName: \"kubernetes.io/projected/bbe8dcdb-21a5-45a2-8102-ab451b7172a7-kube-api-access-l4l8z\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 01:01:06.928036 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.927948 2572 generic.go:356] "Generic (PLEG): container finished" podID="bbe8dcdb-21a5-45a2-8102-ab451b7172a7" containerID="d9feeb15ef61d1ed27afa665333fdefd4cbc93bb49216eaa1d23dabfda274e93" exitCode=0 Jul 30 01:01:06.928036 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.928003 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" Jul 30 01:01:06.928036 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.928019 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" event={"ID":"bbe8dcdb-21a5-45a2-8102-ab451b7172a7","Type":"ContainerDied","Data":"d9feeb15ef61d1ed27afa665333fdefd4cbc93bb49216eaa1d23dabfda274e93"} Jul 30 01:01:06.928336 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.928062 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv" event={"ID":"bbe8dcdb-21a5-45a2-8102-ab451b7172a7","Type":"ContainerDied","Data":"95389ae5dc350ef7f0df66aff579584daa756a91eef26b6995537100f020a5f6"} Jul 30 01:01:06.928336 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.928083 2572 scope.go:117] "RemoveContainer" containerID="d9feeb15ef61d1ed27afa665333fdefd4cbc93bb49216eaa1d23dabfda274e93" Jul 30 01:01:06.936483 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.936467 2572 scope.go:117] "RemoveContainer" containerID="713dfd5eef931190f025b5b2bf7907ef869eb9fbbbd482c229ec36b2ad7ec0ad" Jul 30 01:01:06.943811 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.943788 2572 scope.go:117] "RemoveContainer" containerID="d9feeb15ef61d1ed27afa665333fdefd4cbc93bb49216eaa1d23dabfda274e93" Jul 30 01:01:06.944053 ip-10-0-135-118 kubenswrapper[2572]: E0730 01:01:06.944035 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9feeb15ef61d1ed27afa665333fdefd4cbc93bb49216eaa1d23dabfda274e93\": container with ID starting with d9feeb15ef61d1ed27afa665333fdefd4cbc93bb49216eaa1d23dabfda274e93 not found: ID does not exist" containerID="d9feeb15ef61d1ed27afa665333fdefd4cbc93bb49216eaa1d23dabfda274e93" Jul 30 01:01:06.944158 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.944062 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9feeb15ef61d1ed27afa665333fdefd4cbc93bb49216eaa1d23dabfda274e93"} err="failed to get container status \"d9feeb15ef61d1ed27afa665333fdefd4cbc93bb49216eaa1d23dabfda274e93\": rpc error: code = NotFound desc = could not find container \"d9feeb15ef61d1ed27afa665333fdefd4cbc93bb49216eaa1d23dabfda274e93\": container with ID starting with d9feeb15ef61d1ed27afa665333fdefd4cbc93bb49216eaa1d23dabfda274e93 not found: ID does not exist" Jul 30 01:01:06.944158 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.944080 2572 scope.go:117] "RemoveContainer" containerID="713dfd5eef931190f025b5b2bf7907ef869eb9fbbbd482c229ec36b2ad7ec0ad" Jul 30 01:01:06.944718 ip-10-0-135-118 kubenswrapper[2572]: E0730 01:01:06.944701 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"713dfd5eef931190f025b5b2bf7907ef869eb9fbbbd482c229ec36b2ad7ec0ad\": container with ID starting with 713dfd5eef931190f025b5b2bf7907ef869eb9fbbbd482c229ec36b2ad7ec0ad not found: ID does not exist" containerID="713dfd5eef931190f025b5b2bf7907ef869eb9fbbbd482c229ec36b2ad7ec0ad" Jul 30 01:01:06.944771 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.944722 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713dfd5eef931190f025b5b2bf7907ef869eb9fbbbd482c229ec36b2ad7ec0ad"} err="failed to get container status \"713dfd5eef931190f025b5b2bf7907ef869eb9fbbbd482c229ec36b2ad7ec0ad\": rpc error: code = NotFound desc = could not find container \"713dfd5eef931190f025b5b2bf7907ef869eb9fbbbd482c229ec36b2ad7ec0ad\": container with ID starting with 713dfd5eef931190f025b5b2bf7907ef869eb9fbbbd482c229ec36b2ad7ec0ad not found: ID does not exist" Jul 30 01:01:06.948964 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.948942 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv"] Jul 30 01:01:06.952419 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.952399 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-4ef902-openshift-default-6dcd5d969b-nfwrv"] Jul 30 01:01:06.992827 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:06.992794 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbe8dcdb-21a5-45a2-8102-ab451b7172a7" path="/var/lib/kubelet/pods/bbe8dcdb-21a5-45a2-8102-ab451b7172a7/volumes" Jul 30 01:01:12.099652 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:12.099619 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:01:12.164795 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:12.164768 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:01:12.244611 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:12.244578 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:01:12.301165 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:12.301096 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:01:12.607750 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:12.607708 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:01:18.297322 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:18.297290 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:01:18.354058 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:18.354029 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:01:18.523329 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:18.523294 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph"] Jul 30 01:01:18.523546 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:18.523523 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" podUID="7597295b-dca6-4d83-a443-c12cd95f581e" containerName="istio-proxy" containerID="cri-o://8f8f191e17ee833b3e650df11285f41b7b95a147f286e6ecbcfe12c7cfa33844" gracePeriod=30 Jul 30 01:01:18.548581 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:18.548510 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:01:18.816832 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:18.816748 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:01:23.770946 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:23.770924 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 01:01:23.806745 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:23.806668 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-istio-data\") pod \"7597295b-dca6-4d83-a443-c12cd95f581e\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " Jul 30 01:01:23.806745 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:23.806703 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7597295b-dca6-4d83-a443-c12cd95f581e-istio-token\") pod \"7597295b-dca6-4d83-a443-c12cd95f581e\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " Jul 30 01:01:23.806745 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:23.806733 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmmqz\" (UniqueName: \"kubernetes.io/projected/7597295b-dca6-4d83-a443-c12cd95f581e-kube-api-access-qmmqz\") pod \"7597295b-dca6-4d83-a443-c12cd95f581e\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " Jul 30 01:01:23.807010 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:23.806752 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7597295b-dca6-4d83-a443-c12cd95f581e-istio-podinfo\") pod \"7597295b-dca6-4d83-a443-c12cd95f581e\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " Jul 30 01:01:23.807010 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:23.806774 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-workload-socket\") pod \"7597295b-dca6-4d83-a443-c12cd95f581e\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " Jul 30 01:01:23.807010 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:23.806798 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-istio-envoy\") pod \"7597295b-dca6-4d83-a443-c12cd95f581e\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " Jul 30 01:01:23.807010 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:23.806847 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7597295b-dca6-4d83-a443-c12cd95f581e-istiod-ca-cert\") pod \"7597295b-dca6-4d83-a443-c12cd95f581e\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " Jul 30 01:01:23.807010 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:23.806905 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-credential-socket\") pod \"7597295b-dca6-4d83-a443-c12cd95f581e\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " Jul 30 01:01:23.807010 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:23.806930 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-workload-certs\") pod \"7597295b-dca6-4d83-a443-c12cd95f581e\" (UID: \"7597295b-dca6-4d83-a443-c12cd95f581e\") " Jul 30 01:01:23.807362 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:23.807141 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7597295b-dca6-4d83-a443-c12cd95f581e" (UID: "7597295b-dca6-4d83-a443-c12cd95f581e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:01:23.807362 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:23.807260 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-workload-socket\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 01:01:23.807362 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:23.807321 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7597295b-dca6-4d83-a443-c12cd95f581e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7597295b-dca6-4d83-a443-c12cd95f581e" (UID: "7597295b-dca6-4d83-a443-c12cd95f581e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 01:01:23.807536 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:23.807331 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7597295b-dca6-4d83-a443-c12cd95f581e" (UID: "7597295b-dca6-4d83-a443-c12cd95f581e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:01:23.807536 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:23.807513 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7597295b-dca6-4d83-a443-c12cd95f581e" (UID: "7597295b-dca6-4d83-a443-c12cd95f581e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:01:23.807656 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:23.807624 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-istio-data" (OuterVolumeSpecName: "istio-data") pod "7597295b-dca6-4d83-a443-c12cd95f581e" (UID: "7597295b-dca6-4d83-a443-c12cd95f581e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:01:23.809102 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:23.809068 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7597295b-dca6-4d83-a443-c12cd95f581e-kube-api-access-qmmqz" (OuterVolumeSpecName: "kube-api-access-qmmqz") pod "7597295b-dca6-4d83-a443-c12cd95f581e" (UID: "7597295b-dca6-4d83-a443-c12cd95f581e"). InnerVolumeSpecName "kube-api-access-qmmqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 01:01:23.809248 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:23.809163 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7597295b-dca6-4d83-a443-c12cd95f581e" (UID: "7597295b-dca6-4d83-a443-c12cd95f581e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 01:01:23.809313 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:23.809303 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7597295b-dca6-4d83-a443-c12cd95f581e-istio-token" (OuterVolumeSpecName: "istio-token") pod "7597295b-dca6-4d83-a443-c12cd95f581e" (UID: "7597295b-dca6-4d83-a443-c12cd95f581e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 01:01:23.809503 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:23.809475 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7597295b-dca6-4d83-a443-c12cd95f581e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7597295b-dca6-4d83-a443-c12cd95f581e" (UID: "7597295b-dca6-4d83-a443-c12cd95f581e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 01:01:23.907851 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:23.907817 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmmqz\" (UniqueName: \"kubernetes.io/projected/7597295b-dca6-4d83-a443-c12cd95f581e-kube-api-access-qmmqz\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 01:01:23.907851 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:23.907845 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7597295b-dca6-4d83-a443-c12cd95f581e-istio-podinfo\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 01:01:23.907851 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:23.907855 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-istio-envoy\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 01:01:23.908085 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:23.907863 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7597295b-dca6-4d83-a443-c12cd95f581e-istiod-ca-cert\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 01:01:23.908085 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:23.907872 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-credential-socket\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 01:01:23.908085 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:23.907881 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-workload-certs\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 01:01:23.908085 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:23.907889 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7597295b-dca6-4d83-a443-c12cd95f581e-istio-data\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 01:01:23.908085 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:23.907896 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7597295b-dca6-4d83-a443-c12cd95f581e-istio-token\") on node \"ip-10-0-135-118.ec2.internal\" DevicePath \"\"" Jul 30 01:01:24.004090 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:24.004057 2572 generic.go:356] "Generic (PLEG): container finished" podID="7597295b-dca6-4d83-a443-c12cd95f581e" containerID="8f8f191e17ee833b3e650df11285f41b7b95a147f286e6ecbcfe12c7cfa33844" exitCode=0 Jul 30 01:01:24.004271 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:24.004140 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" Jul 30 01:01:24.004271 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:24.004139 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" event={"ID":"7597295b-dca6-4d83-a443-c12cd95f581e","Type":"ContainerDied","Data":"8f8f191e17ee833b3e650df11285f41b7b95a147f286e6ecbcfe12c7cfa33844"} Jul 30 01:01:24.004369 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:24.004279 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph" event={"ID":"7597295b-dca6-4d83-a443-c12cd95f581e","Type":"ContainerDied","Data":"38be4e973dbc847d17375ec3767a8dbb73f84ee60dbec524e8a739a2310c17d7"} Jul 30 01:01:24.004369 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:24.004308 2572 scope.go:117] "RemoveContainer" containerID="8f8f191e17ee833b3e650df11285f41b7b95a147f286e6ecbcfe12c7cfa33844" Jul 30 01:01:24.012704 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:24.012688 2572 scope.go:117] "RemoveContainer" containerID="de6d4b46a40113da7bdd78f8d8de006a1863a1bfe2463e0ce3163f04e7dbfc4a" Jul 30 01:01:24.019780 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:24.019763 2572 scope.go:117] "RemoveContainer" containerID="8f8f191e17ee833b3e650df11285f41b7b95a147f286e6ecbcfe12c7cfa33844" Jul 30 01:01:24.020034 ip-10-0-135-118 kubenswrapper[2572]: E0730 01:01:24.020016 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f8f191e17ee833b3e650df11285f41b7b95a147f286e6ecbcfe12c7cfa33844\": container with ID starting with 8f8f191e17ee833b3e650df11285f41b7b95a147f286e6ecbcfe12c7cfa33844 not found: ID does not exist" containerID="8f8f191e17ee833b3e650df11285f41b7b95a147f286e6ecbcfe12c7cfa33844" Jul 30 01:01:24.020163 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:24.020045 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f8f191e17ee833b3e650df11285f41b7b95a147f286e6ecbcfe12c7cfa33844"} err="failed to get container status \"8f8f191e17ee833b3e650df11285f41b7b95a147f286e6ecbcfe12c7cfa33844\": rpc error: code = NotFound desc = could not find container \"8f8f191e17ee833b3e650df11285f41b7b95a147f286e6ecbcfe12c7cfa33844\": container with ID starting with 8f8f191e17ee833b3e650df11285f41b7b95a147f286e6ecbcfe12c7cfa33844 not found: ID does not exist" Jul 30 01:01:24.020163 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:24.020068 2572 scope.go:117] "RemoveContainer" containerID="de6d4b46a40113da7bdd78f8d8de006a1863a1bfe2463e0ce3163f04e7dbfc4a" Jul 30 01:01:24.020368 ip-10-0-135-118 kubenswrapper[2572]: E0730 01:01:24.020349 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de6d4b46a40113da7bdd78f8d8de006a1863a1bfe2463e0ce3163f04e7dbfc4a\": container with ID starting with de6d4b46a40113da7bdd78f8d8de006a1863a1bfe2463e0ce3163f04e7dbfc4a not found: ID does not exist" containerID="de6d4b46a40113da7bdd78f8d8de006a1863a1bfe2463e0ce3163f04e7dbfc4a" Jul 30 01:01:24.020417 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:24.020374 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de6d4b46a40113da7bdd78f8d8de006a1863a1bfe2463e0ce3163f04e7dbfc4a"} err="failed to get container status \"de6d4b46a40113da7bdd78f8d8de006a1863a1bfe2463e0ce3163f04e7dbfc4a\": rpc error: code = NotFound desc = could not find container \"de6d4b46a40113da7bdd78f8d8de006a1863a1bfe2463e0ce3163f04e7dbfc4a\": container with ID starting with de6d4b46a40113da7bdd78f8d8de006a1863a1bfe2463e0ce3163f04e7dbfc4a not found: ID does not exist" Jul 30 01:01:24.024600 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:24.024579 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph"] Jul 30 01:01:24.029164 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:24.029140 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-0b2dae-openshift-default-96d588c44-vn8ph"] Jul 30 01:01:24.740385 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:24.740352 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-76889fd47d-8tvfw_91baeab4-4ae3-428a-a5e8-96ff22f0773e/manager/0.log" Jul 30 01:01:24.991737 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:24.991662 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7597295b-dca6-4d83-a443-c12cd95f581e" path="/var/lib/kubelet/pods/7597295b-dca6-4d83-a443-c12cd95f581e/volumes" Jul 30 01:01:25.340811 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:25.340788 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-655d97b687-l8fnr_b446af54-7c30-437b-9faa-7e17c067316e/copy-manifests/0.log" Jul 30 01:01:25.347235 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:25.347212 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-655d97b687-l8fnr_b446af54-7c30-437b-9faa-7e17c067316e/manager/0.log" Jul 30 01:01:25.683016 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:25.682929 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-8667dbffc5-nm7rm_d70e206a-eb89-4d5b-83e9-e08ddbdd8d35/maas-api/0.log" Jul 30 01:01:26.111008 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:26.110982 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-4dmtb_7a9e81dd-5d22-4171-8734-6e0b8aafe115/postgres/0.log" Jul 30 01:01:28.526896 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:28.526856 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8_937f26e2-fce2-4ab8-8c74-a71670a333cd/istio-proxy/0.log" Jul 30 01:01:28.643610 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:28.643584 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-w62gl_5826506a-70dc-4388-923e-cb7f5df65924/discovery/0.log" Jul 30 01:01:29.779618 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:29.779587 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-nbddf_b2a0a320-e247-4d6d-9230-de3819c88c54/main/0.log" Jul 30 01:01:30.112516 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:30.112474 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-d5cpp_924002eb-17c0-426e-b65a-39519ffe69d5/main/0.log" Jul 30 01:01:35.989343 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:35.989311 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:01:36.054082 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:36.054051 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" limit=1024 Jul 30 01:01:39.137070 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:39.137032 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-77hgn_46aec834-30bb-4d73-b1b7-f033193e6e99/global-pull-secret-syncer/0.log" Jul 30 01:01:39.286963 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:39.286926 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-g7zvw_f1f125ac-f373-4ad9-8734-263caec79393/konnectivity-agent/0.log" Jul 30 01:01:39.434095 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:39.434006 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-118.ec2.internal_60d5ee2ac122fad92c39ced53363933e/haproxy/0.log" Jul 30 01:01:47.077257 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:47.077222 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8h6k9_1c121fea-270e-4763-aff7-53348028373d/node-exporter/0.log" Jul 30 01:01:47.102052 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:47.102026 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8h6k9_1c121fea-270e-4763-aff7-53348028373d/kube-rbac-proxy/0.log" Jul 30 01:01:47.129470 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:47.129445 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8h6k9_1c121fea-270e-4763-aff7-53348028373d/init-textfile/0.log" Jul 30 01:01:47.550718 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:47.550621 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-lmwz5_63ffb722-7c47-41f7-8a27-96664c33e1e5/prometheus-operator/0.log" Jul 30 01:01:47.569018 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:47.568996 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-lmwz5_63ffb722-7c47-41f7-8a27-96664c33e1e5/kube-rbac-proxy/0.log" Jul 30 01:01:49.715607 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.715566 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mf94r/perf-node-gather-daemonset-fmqcc"] Jul 30 01:01:49.716018 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.715867 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7597295b-dca6-4d83-a443-c12cd95f581e" containerName="istio-proxy" Jul 30 01:01:49.716018 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.715877 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7597295b-dca6-4d83-a443-c12cd95f581e" containerName="istio-proxy" Jul 30 01:01:49.716018 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.715884 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fdc83fa-b7d8-4c85-a844-7a0aa2494abe" containerName="istio-proxy" Jul 30 01:01:49.716018 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.715890 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fdc83fa-b7d8-4c85-a844-7a0aa2494abe" containerName="istio-proxy" Jul 30 01:01:49.716018 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.715900 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33e445d9-72a1-45b0-aa61-449f8c6f66f5" containerName="maas-api" Jul 30 01:01:49.716018 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.715907 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="33e445d9-72a1-45b0-aa61-449f8c6f66f5" containerName="maas-api" Jul 30 01:01:49.716018 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.715918 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4" containerName="maas-api" Jul 30 01:01:49.716018 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.715923 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4" containerName="maas-api" Jul 30 01:01:49.716018 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.715930 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fdc83fa-b7d8-4c85-a844-7a0aa2494abe" containerName="istio-proxy" Jul 30 01:01:49.716018 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.715936 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fdc83fa-b7d8-4c85-a844-7a0aa2494abe" containerName="istio-proxy" Jul 30 01:01:49.716018 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.715944 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbe8dcdb-21a5-45a2-8102-ab451b7172a7" containerName="istio-proxy" Jul 30 01:01:49.716018 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.715951 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbe8dcdb-21a5-45a2-8102-ab451b7172a7" containerName="istio-proxy" Jul 30 01:01:49.716018 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.715962 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbe8dcdb-21a5-45a2-8102-ab451b7172a7" containerName="istio-proxy" Jul 30 01:01:49.716018 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.715967 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbe8dcdb-21a5-45a2-8102-ab451b7172a7" containerName="istio-proxy" Jul 30 01:01:49.716018 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.715973 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7597295b-dca6-4d83-a443-c12cd95f581e" containerName="istio-proxy" Jul 30 01:01:49.716018 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.715978 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7597295b-dca6-4d83-a443-c12cd95f581e" containerName="istio-proxy" Jul 30 01:01:49.716018 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.716026 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7597295b-dca6-4d83-a443-c12cd95f581e" containerName="istio-proxy" Jul 30 01:01:49.716770 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.716034 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6b6080a-9a6e-4513-aa45-6c7ea6b8d6a4" containerName="maas-api" Jul 30 01:01:49.716770 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.716041 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fdc83fa-b7d8-4c85-a844-7a0aa2494abe" containerName="istio-proxy" Jul 30 01:01:49.716770 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.716046 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fdc83fa-b7d8-4c85-a844-7a0aa2494abe" containerName="istio-proxy" Jul 30 01:01:49.716770 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.716053 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbe8dcdb-21a5-45a2-8102-ab451b7172a7" containerName="istio-proxy" Jul 30 01:01:49.716770 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.716058 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbe8dcdb-21a5-45a2-8102-ab451b7172a7" containerName="istio-proxy" Jul 30 01:01:49.716770 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.716065 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="33e445d9-72a1-45b0-aa61-449f8c6f66f5" containerName="maas-api" Jul 30 01:01:49.716770 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.716070 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7597295b-dca6-4d83-a443-c12cd95f581e" containerName="istio-proxy" Jul 30 01:01:49.719135 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.719095 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mf94r/perf-node-gather-daemonset-fmqcc" Jul 30 01:01:49.721699 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.721669 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mf94r\"/\"openshift-service-ca.crt\"" Jul 30 01:01:49.721829 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.721738 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mf94r\"/\"default-dockercfg-crx64\"" Jul 30 01:01:49.722694 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.722679 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mf94r\"/\"kube-root-ca.crt\"" Jul 30 01:01:49.729484 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.729459 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mf94r/perf-node-gather-daemonset-fmqcc"] Jul 30 01:01:49.832082 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.832043 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/26ac8732-f528-45d7-b492-54a0f4811eba-podres\") pod \"perf-node-gather-daemonset-fmqcc\" (UID: \"26ac8732-f528-45d7-b492-54a0f4811eba\") " pod="openshift-must-gather-mf94r/perf-node-gather-daemonset-fmqcc" Jul 30 01:01:49.832082 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.832089 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26ac8732-f528-45d7-b492-54a0f4811eba-lib-modules\") pod \"perf-node-gather-daemonset-fmqcc\" (UID: \"26ac8732-f528-45d7-b492-54a0f4811eba\") " pod="openshift-must-gather-mf94r/perf-node-gather-daemonset-fmqcc" Jul 30 01:01:49.832345 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.832147 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rhvg\" (UniqueName: \"kubernetes.io/projected/26ac8732-f528-45d7-b492-54a0f4811eba-kube-api-access-9rhvg\") pod \"perf-node-gather-daemonset-fmqcc\" (UID: \"26ac8732-f528-45d7-b492-54a0f4811eba\") " pod="openshift-must-gather-mf94r/perf-node-gather-daemonset-fmqcc" Jul 30 01:01:49.832345 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.832246 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/26ac8732-f528-45d7-b492-54a0f4811eba-proc\") pod \"perf-node-gather-daemonset-fmqcc\" (UID: \"26ac8732-f528-45d7-b492-54a0f4811eba\") " pod="openshift-must-gather-mf94r/perf-node-gather-daemonset-fmqcc" Jul 30 01:01:49.832345 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.832285 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26ac8732-f528-45d7-b492-54a0f4811eba-sys\") pod \"perf-node-gather-daemonset-fmqcc\" (UID: \"26ac8732-f528-45d7-b492-54a0f4811eba\") " pod="openshift-must-gather-mf94r/perf-node-gather-daemonset-fmqcc" Jul 30 01:01:49.933226 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.933184 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/26ac8732-f528-45d7-b492-54a0f4811eba-proc\") pod \"perf-node-gather-daemonset-fmqcc\" (UID: \"26ac8732-f528-45d7-b492-54a0f4811eba\") " pod="openshift-must-gather-mf94r/perf-node-gather-daemonset-fmqcc" Jul 30 01:01:49.933226 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.933228 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26ac8732-f528-45d7-b492-54a0f4811eba-sys\") pod \"perf-node-gather-daemonset-fmqcc\" (UID: \"26ac8732-f528-45d7-b492-54a0f4811eba\") " pod="openshift-must-gather-mf94r/perf-node-gather-daemonset-fmqcc" Jul 30 01:01:49.933449 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.933255 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/26ac8732-f528-45d7-b492-54a0f4811eba-podres\") pod \"perf-node-gather-daemonset-fmqcc\" (UID: \"26ac8732-f528-45d7-b492-54a0f4811eba\") " pod="openshift-must-gather-mf94r/perf-node-gather-daemonset-fmqcc" Jul 30 01:01:49.933449 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.933275 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26ac8732-f528-45d7-b492-54a0f4811eba-lib-modules\") pod \"perf-node-gather-daemonset-fmqcc\" (UID: \"26ac8732-f528-45d7-b492-54a0f4811eba\") " pod="openshift-must-gather-mf94r/perf-node-gather-daemonset-fmqcc" Jul 30 01:01:49.933449 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.933311 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rhvg\" (UniqueName: \"kubernetes.io/projected/26ac8732-f528-45d7-b492-54a0f4811eba-kube-api-access-9rhvg\") pod \"perf-node-gather-daemonset-fmqcc\" (UID: \"26ac8732-f528-45d7-b492-54a0f4811eba\") " pod="openshift-must-gather-mf94r/perf-node-gather-daemonset-fmqcc" Jul 30 01:01:49.933449 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.933332 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/26ac8732-f528-45d7-b492-54a0f4811eba-proc\") pod \"perf-node-gather-daemonset-fmqcc\" (UID: \"26ac8732-f528-45d7-b492-54a0f4811eba\") " pod="openshift-must-gather-mf94r/perf-node-gather-daemonset-fmqcc" Jul 30 01:01:49.933449 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.933332 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26ac8732-f528-45d7-b492-54a0f4811eba-sys\") pod \"perf-node-gather-daemonset-fmqcc\" (UID: \"26ac8732-f528-45d7-b492-54a0f4811eba\") " pod="openshift-must-gather-mf94r/perf-node-gather-daemonset-fmqcc" Jul 30 01:01:49.933449 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.933399 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/26ac8732-f528-45d7-b492-54a0f4811eba-podres\") pod \"perf-node-gather-daemonset-fmqcc\" (UID: \"26ac8732-f528-45d7-b492-54a0f4811eba\") " pod="openshift-must-gather-mf94r/perf-node-gather-daemonset-fmqcc" Jul 30 01:01:49.933449 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.933419 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26ac8732-f528-45d7-b492-54a0f4811eba-lib-modules\") pod \"perf-node-gather-daemonset-fmqcc\" (UID: \"26ac8732-f528-45d7-b492-54a0f4811eba\") " pod="openshift-must-gather-mf94r/perf-node-gather-daemonset-fmqcc" Jul 30 01:01:49.941644 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:49.941602 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rhvg\" (UniqueName: \"kubernetes.io/projected/26ac8732-f528-45d7-b492-54a0f4811eba-kube-api-access-9rhvg\") pod \"perf-node-gather-daemonset-fmqcc\" (UID: \"26ac8732-f528-45d7-b492-54a0f4811eba\") " pod="openshift-must-gather-mf94r/perf-node-gather-daemonset-fmqcc" Jul 30 01:01:50.030201 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:50.030087 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mf94r/perf-node-gather-daemonset-fmqcc" Jul 30 01:01:50.167349 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:50.167185 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mf94r/perf-node-gather-daemonset-fmqcc"] Jul 30 01:01:50.169978 ip-10-0-135-118 kubenswrapper[2572]: W0730 01:01:50.169945 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod26ac8732_f528_45d7_b492_54a0f4811eba.slice/crio-6ff8d7ec061c9ea4ee4c50a5da2358795422992d531d033acf9cbb2df02366c7 WatchSource:0}: Error finding container 6ff8d7ec061c9ea4ee4c50a5da2358795422992d531d033acf9cbb2df02366c7: Status 404 returned error can't find the container with id 6ff8d7ec061c9ea4ee4c50a5da2358795422992d531d033acf9cbb2df02366c7 Jul 30 01:01:51.113403 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:51.113369 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mf94r/perf-node-gather-daemonset-fmqcc" event={"ID":"26ac8732-f528-45d7-b492-54a0f4811eba","Type":"ContainerStarted","Data":"8ca92b8ef4e97f7be64d6147ab209c3db95af2f67cf11674a5e0ec246d6c8669"} Jul 30 01:01:51.113799 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:51.113425 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mf94r/perf-node-gather-daemonset-fmqcc" event={"ID":"26ac8732-f528-45d7-b492-54a0f4811eba","Type":"ContainerStarted","Data":"6ff8d7ec061c9ea4ee4c50a5da2358795422992d531d033acf9cbb2df02366c7"} Jul 30 01:01:51.113799 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:51.113455 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mf94r/perf-node-gather-daemonset-fmqcc" Jul 30 01:01:51.128912 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:51.128860 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mf94r/perf-node-gather-daemonset-fmqcc" podStartSLOduration=2.128843385 podStartE2EDuration="2.128843385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 01:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 01:01:51.128266984 +0000 UTC m=+3598.631525243" watchObservedRunningTime="2026-07-30 01:01:51.128843385 +0000 UTC m=+3598.632101644" Jul 30 01:01:51.820875 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:51.820847 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tb6c5_34dd2a14-f850-4a05-ab6b-9c0c0196ed0a/dns/0.log" Jul 30 01:01:51.839690 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:51.839664 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tb6c5_34dd2a14-f850-4a05-ab6b-9c0c0196ed0a/kube-rbac-proxy/0.log" Jul 30 01:01:51.912101 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:51.912074 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zsmpb_6bf193c2-d465-4e6c-abaf-afc0f08815fa/dns-node-resolver/0.log" Jul 30 01:01:52.444003 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:52.443950 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-65995cc6bf-9kq6s_0a96f39c-93c9-4cc7-b2c2-e189529a15f4/registry/0.log" Jul 30 01:01:52.518073 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:52.518044 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lbgfr_aa41535c-676c-430a-9151-7257622c4076/node-ca/0.log" Jul 30 01:01:53.437447 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:53.437400 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8_937f26e2-fce2-4ab8-8c74-a71670a333cd/istio-proxy/0.log" Jul 30 01:01:53.577488 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:53.577456 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-w62gl_5826506a-70dc-4388-923e-cb7f5df65924/discovery/0.log" Jul 30 01:01:54.652588 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:54.652560 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mksd5_b9b8edba-eb6a-4906-97a4-4eb273367aed/serve-healthcheck-canary/0.log" Jul 30 01:01:55.260267 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:55.260235 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nx2v9_146c3400-fc20-48bb-a456-e7b4f53f406f/kube-rbac-proxy/0.log" Jul 30 01:01:55.280318 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:55.280293 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nx2v9_146c3400-fc20-48bb-a456-e7b4f53f406f/exporter/0.log" Jul 30 01:01:55.302100 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:55.302071 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nx2v9_146c3400-fc20-48bb-a456-e7b4f53f406f/extractor/0.log" Jul 30 01:01:57.128751 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:57.128725 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mf94r/perf-node-gather-daemonset-fmqcc" Jul 30 01:01:57.999695 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:57.999658 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-76889fd47d-8tvfw_91baeab4-4ae3-428a-a5e8-96ff22f0773e/manager/0.log" Jul 30 01:01:58.378462 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:58.378435 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-655d97b687-l8fnr_b446af54-7c30-437b-9faa-7e17c067316e/manager/0.log" Jul 30 01:01:58.400999 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:58.400967 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-655d97b687-l8fnr_b446af54-7c30-437b-9faa-7e17c067316e/copy-manifests/0.log" Jul 30 01:01:59.842477 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:59.842452 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5977989bd7-9xvmv_e88b0711-0653-42eb-a6e0-1272d5e62b8f/manager/0.log" Jul 30 01:01:59.901061 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:01:59.901016 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8_937f26e2-fce2-4ab8-8c74-a71670a333cd/istio-proxy/0.log" Jul 30 01:02:00.041429 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:02:00.041397 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-w62gl_5826506a-70dc-4388-923e-cb7f5df65924/discovery/0.log" Jul 30 01:02:01.578390 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:02:01.578362 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cfq4n8_937f26e2-fce2-4ab8-8c74-a71670a333cd/istio-proxy/0.log" Jul 30 01:02:01.714810 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:02:01.714783 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-w62gl_5826506a-70dc-4388-923e-cb7f5df65924/discovery/0.log" Jul 30 01:02:06.186552 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:02:06.186517 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lg8hw_5bfc194e-138e-4335-87f4-f6837c8f2e49/kube-multus-additional-cni-plugins/0.log" Jul 30 01:02:06.206058 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:02:06.206035 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lg8hw_5bfc194e-138e-4335-87f4-f6837c8f2e49/egress-router-binary-copy/0.log" Jul 30 01:02:06.233235 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:02:06.233199 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lg8hw_5bfc194e-138e-4335-87f4-f6837c8f2e49/cni-plugins/0.log" Jul 30 01:02:06.252864 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:02:06.252842 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lg8hw_5bfc194e-138e-4335-87f4-f6837c8f2e49/bond-cni-plugin/0.log" Jul 30 01:02:06.273189 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:02:06.273161 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lg8hw_5bfc194e-138e-4335-87f4-f6837c8f2e49/routeoverride-cni/0.log" Jul 30 01:02:06.294893 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:02:06.294848 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lg8hw_5bfc194e-138e-4335-87f4-f6837c8f2e49/whereabouts-cni-bincopy/0.log" Jul 30 01:02:06.314600 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:02:06.314570 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lg8hw_5bfc194e-138e-4335-87f4-f6837c8f2e49/whereabouts-cni/0.log" Jul 30 01:02:06.483531 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:02:06.483442 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r2jkk_afff0c7f-4d59-4a4f-bc4e-2dcdb2521463/kube-multus/0.log" Jul 30 01:02:06.687337 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:02:06.687304 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nk989_a38e0c1c-7133-40ff-9ba4-9a066da5c41e/network-metrics-daemon/0.log" Jul 30 01:02:06.707978 ip-10-0-135-118 kubenswrapper[2572]: I0730 01:02:06.707950 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nk989_a38e0c1c-7133-40ff-9ba4-9a066da5c41e/kube-rbac-proxy/0.log"