Jul 08 02:56:33.685427 ip-10-0-128-70 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 08 02:56:33.685438 ip-10-0-128-70 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 08 02:56:33.685445 ip-10-0-128-70 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 08 02:56:33.685657 ip-10-0-128-70 systemd[1]: Failed to start Kubernetes Kubelet. Jul 08 02:56:43.909020 ip-10-0-128-70 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 08 02:56:43.909040 ip-10-0-128-70 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 8184b1c1f22b45b9b0763efc860aca4f -- Jul 08 02:58:45.231267 ip-10-0-128-70 systemd[1]: Starting Kubernetes Kubelet... Jul 08 02:58:45.615412 ip-10-0-128-70 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 02:58:45.615412 ip-10-0-128-70 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 08 02:58:45.615412 ip-10-0-128-70 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 02:58:45.615412 ip-10-0-128-70 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 08 02:58:45.615412 ip-10-0-128-70 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 02:58:45.617971 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.617870 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 08 02:58:45.621469 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621450 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 02:58:45.621469 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621469 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 02:58:45.621469 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621472 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 02:58:45.621569 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621476 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 02:58:45.621569 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621479 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 02:58:45.621569 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621482 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 02:58:45.621569 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621485 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 02:58:45.621569 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621487 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 02:58:45.621569 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621494 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 02:58:45.621569 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621497 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 02:58:45.621569 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621501 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 02:58:45.621569 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621505 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 02:58:45.621569 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621508 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 02:58:45.621569 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621512 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 02:58:45.621569 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621514 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 02:58:45.621569 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621517 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 02:58:45.621569 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621520 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 02:58:45.621569 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621523 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 02:58:45.621569 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621526 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 02:58:45.621569 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621529 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 02:58:45.621569 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621532 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 02:58:45.621569 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621534 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 02:58:45.621569 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621536 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 02:58:45.622061 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621539 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 02:58:45.622061 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621542 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 02:58:45.622061 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621544 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 02:58:45.622061 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621547 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 02:58:45.622061 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621549 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 02:58:45.622061 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621552 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 02:58:45.622061 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621554 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 02:58:45.622061 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621557 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 02:58:45.622061 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621559 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 02:58:45.622061 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621561 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 02:58:45.622061 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621564 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 02:58:45.622061 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621566 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 02:58:45.622061 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621569 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 02:58:45.622061 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621572 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 02:58:45.622061 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621575 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 02:58:45.622061 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621579 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 02:58:45.622061 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621581 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 02:58:45.622061 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621584 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 02:58:45.622061 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621587 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 02:58:45.622061 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621589 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 02:58:45.622582 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621592 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 02:58:45.622582 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621594 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 02:58:45.622582 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621597 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 02:58:45.622582 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621599 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 02:58:45.622582 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621601 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 02:58:45.622582 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621604 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 02:58:45.622582 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621607 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 02:58:45.622582 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621609 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 02:58:45.622582 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621612 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 02:58:45.622582 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621614 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 02:58:45.622582 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621619 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 02:58:45.622582 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621623 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 02:58:45.622582 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621626 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 02:58:45.622582 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621628 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 02:58:45.622582 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621631 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 02:58:45.622582 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621633 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 02:58:45.622582 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621636 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 02:58:45.622582 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621639 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 02:58:45.622582 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621641 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 02:58:45.622582 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621643 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 08 02:58:45.623082 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621646 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 02:58:45.623082 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621648 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 02:58:45.623082 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621651 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 02:58:45.623082 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621653 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 02:58:45.623082 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621656 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 02:58:45.623082 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621660 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 02:58:45.623082 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621663 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 02:58:45.623082 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621665 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 02:58:45.623082 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621669 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 02:58:45.623082 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621672 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 02:58:45.623082 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621675 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 02:58:45.623082 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621677 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 02:58:45.623082 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621682 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 02:58:45.623082 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621686 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 02:58:45.623082 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621689 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 02:58:45.623082 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621691 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 02:58:45.623082 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621694 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 02:58:45.623082 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621697 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 02:58:45.623082 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621700 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 02:58:45.623529 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621702 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 02:58:45.623529 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621705 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 02:58:45.623529 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621707 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 02:58:45.623529 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621709 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 02:58:45.623529 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621712 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 02:58:45.623529 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621714 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 02:58:45.623529 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621717 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 02:58:45.623529 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621721 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 02:58:45.623529 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621724 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 02:58:45.623529 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621726 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 02:58:45.623529 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621729 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 02:58:45.623529 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621732 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 02:58:45.623529 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621735 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 02:58:45.623529 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621737 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 02:58:45.623529 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621740 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 02:58:45.623529 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621742 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 02:58:45.623529 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.621745 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 02:58:45.623529 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623451 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 02:58:45.623529 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623459 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 02:58:45.624010 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623463 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 02:58:45.624010 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623466 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 02:58:45.624010 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623469 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 02:58:45.624010 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623472 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 02:58:45.624010 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623475 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 02:58:45.624010 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623478 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 02:58:45.624010 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623481 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 02:58:45.624010 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623484 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 02:58:45.624010 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623486 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 02:58:45.624010 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623489 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 02:58:45.624010 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623492 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 02:58:45.624010 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623496 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 02:58:45.624010 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623500 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 02:58:45.624010 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623503 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 02:58:45.624010 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623506 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 02:58:45.624010 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623509 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 02:58:45.624010 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623512 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 02:58:45.624010 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623514 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 02:58:45.624010 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623517 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 02:58:45.624473 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623519 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 02:58:45.624473 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623522 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 02:58:45.624473 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623525 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 02:58:45.624473 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623527 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 02:58:45.624473 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623530 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 02:58:45.624473 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623533 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 02:58:45.624473 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623536 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 02:58:45.624473 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623538 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 02:58:45.624473 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623541 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 02:58:45.624473 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623543 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 02:58:45.624473 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623546 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 02:58:45.624473 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623548 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 02:58:45.624473 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623551 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 02:58:45.624473 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623554 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 02:58:45.624473 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623557 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 02:58:45.624473 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623560 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 02:58:45.624473 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623563 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 02:58:45.624473 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623566 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 08 02:58:45.624473 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623569 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 02:58:45.624473 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623571 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 02:58:45.624473 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623573 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 02:58:45.625163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623576 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 02:58:45.625163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623578 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 02:58:45.625163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623581 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 02:58:45.625163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623583 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 02:58:45.625163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623586 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 02:58:45.625163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623588 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 02:58:45.625163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623591 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 02:58:45.625163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623594 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 02:58:45.625163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623596 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 02:58:45.625163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623599 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 02:58:45.625163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623601 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 02:58:45.625163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623604 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 02:58:45.625163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623607 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 02:58:45.625163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623609 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 02:58:45.625163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623612 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 02:58:45.625163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623614 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 02:58:45.625163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623616 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 02:58:45.625163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623619 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 02:58:45.625163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623621 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 02:58:45.625163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623623 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 02:58:45.625651 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623626 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 02:58:45.625651 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623628 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 02:58:45.625651 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623631 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 02:58:45.625651 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623633 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 02:58:45.625651 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623635 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 02:58:45.625651 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623638 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 02:58:45.625651 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623641 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 02:58:45.625651 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623643 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 02:58:45.625651 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623646 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 02:58:45.625651 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623649 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 02:58:45.625651 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623651 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 02:58:45.625651 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623654 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 02:58:45.625651 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623656 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 02:58:45.625651 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623659 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 02:58:45.625651 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623663 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 02:58:45.625651 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623665 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 02:58:45.625651 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623667 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 02:58:45.625651 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623670 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 02:58:45.625651 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623672 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 02:58:45.625651 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623675 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 02:58:45.626163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623678 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 02:58:45.626163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623680 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 02:58:45.626163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623682 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 02:58:45.626163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623685 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 02:58:45.626163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623687 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 02:58:45.626163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623690 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 02:58:45.626163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623692 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 02:58:45.626163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623696 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 02:58:45.626163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623699 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 02:58:45.626163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623701 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 02:58:45.626163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623704 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 02:58:45.626163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623707 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 02:58:45.626163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623710 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 02:58:45.626163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623712 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 02:58:45.626163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623715 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 02:58:45.626163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623717 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 02:58:45.626163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.623720 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 02:58:45.626163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624120 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 02:58:45.626163 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624126 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 02:58:45.626631 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624128 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 02:58:45.626631 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624131 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 02:58:45.626631 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624134 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 02:58:45.626631 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624137 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 02:58:45.626631 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624139 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 02:58:45.626631 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624142 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 02:58:45.626631 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624145 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 02:58:45.626631 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624147 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 02:58:45.626631 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624149 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 02:58:45.626631 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624154 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 02:58:45.626631 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624157 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 02:58:45.626631 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624160 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 02:58:45.626631 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624163 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 02:58:45.626631 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624166 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 02:58:45.626631 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624169 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 02:58:45.626631 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624173 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 02:58:45.626631 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624175 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 02:58:45.626631 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624178 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 02:58:45.626631 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624180 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 02:58:45.627124 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624183 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 02:58:45.627124 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624185 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 02:58:45.627124 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624188 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 02:58:45.627124 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624190 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 02:58:45.627124 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624193 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 02:58:45.627124 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624195 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 02:58:45.627124 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624198 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 02:58:45.627124 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624200 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 02:58:45.627124 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624203 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 02:58:45.627124 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624205 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 02:58:45.627124 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624208 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 02:58:45.627124 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624211 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 02:58:45.627124 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624213 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 02:58:45.627124 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624216 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 02:58:45.627124 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624218 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 02:58:45.627124 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624221 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 02:58:45.627124 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624223 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 02:58:45.627124 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624226 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 02:58:45.627124 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624228 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 02:58:45.627124 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624231 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 02:58:45.627600 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624233 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 02:58:45.627600 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624235 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 02:58:45.627600 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624238 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 02:58:45.627600 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624240 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 02:58:45.627600 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624243 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 02:58:45.627600 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624245 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 02:58:45.627600 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624248 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 02:58:45.627600 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624250 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 02:58:45.627600 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624252 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 02:58:45.627600 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624255 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 02:58:45.627600 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624258 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 02:58:45.627600 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624260 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 02:58:45.627600 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624263 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 02:58:45.627600 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624265 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 02:58:45.627600 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624268 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 02:58:45.627600 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624270 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 02:58:45.627600 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624272 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 02:58:45.627600 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624275 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 02:58:45.627600 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624277 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 02:58:45.627600 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624280 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 02:58:45.627600 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624282 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 02:58:45.628130 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624284 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 02:58:45.628130 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624287 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 02:58:45.628130 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624289 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 02:58:45.628130 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624292 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 02:58:45.628130 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624295 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 02:58:45.628130 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624298 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 02:58:45.628130 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624300 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 02:58:45.628130 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624302 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 02:58:45.628130 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624305 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 02:58:45.628130 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624307 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 02:58:45.628130 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624310 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 02:58:45.628130 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624312 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 02:58:45.628130 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624314 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 02:58:45.628130 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624316 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 02:58:45.628130 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624319 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 02:58:45.628130 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624324 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 02:58:45.628130 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624326 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 02:58:45.628130 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624329 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 02:58:45.628130 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624331 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 02:58:45.628584 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624334 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 02:58:45.628584 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624336 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 02:58:45.628584 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624339 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 02:58:45.628584 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624342 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 02:58:45.628584 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624344 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 02:58:45.628584 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624347 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 02:58:45.628584 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624349 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 02:58:45.628584 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624353 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 02:58:45.628584 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624356 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 08 02:58:45.628584 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624359 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 02:58:45.628584 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624362 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 02:58:45.628584 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624364 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 02:58:45.628584 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624367 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 02:58:45.628584 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624370 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 02:58:45.628584 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624372 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 02:58:45.628584 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624375 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 02:58:45.628584 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624378 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 02:58:45.628584 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.624380 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 02:58:45.628584 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624456 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 08 02:58:45.628584 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624463 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 08 02:58:45.629093 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624470 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 08 02:58:45.629093 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624474 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 08 02:58:45.629093 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624479 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 08 02:58:45.629093 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624483 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 08 02:58:45.629093 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624487 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 08 02:58:45.629093 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624492 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 08 02:58:45.629093 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624495 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 08 02:58:45.629093 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624498 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 08 02:58:45.629093 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624506 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 08 02:58:45.629093 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624510 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 08 02:58:45.629093 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624513 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 08 02:58:45.629093 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624516 2564 flags.go:64] FLAG: --cgroup-root="" Jul 08 02:58:45.629093 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624518 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 08 02:58:45.629093 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624521 2564 flags.go:64] FLAG: --client-ca-file="" Jul 08 02:58:45.629093 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624524 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 08 02:58:45.629093 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624527 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 08 02:58:45.629093 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624532 2564 flags.go:64] FLAG: --cluster-domain="" Jul 08 02:58:45.629093 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624535 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 08 02:58:45.629093 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624538 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 08 02:58:45.629093 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624541 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 08 02:58:45.629093 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624544 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 08 02:58:45.629093 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624548 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 08 02:58:45.629093 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624551 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 08 02:58:45.629639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624555 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 08 02:58:45.629639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624558 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 08 02:58:45.629639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624561 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 08 02:58:45.629639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624564 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 08 02:58:45.629639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624567 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 08 02:58:45.629639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624570 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 08 02:58:45.629639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624573 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 08 02:58:45.629639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624577 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 08 02:58:45.629639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624580 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 08 02:58:45.629639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624583 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 08 02:58:45.629639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624586 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 08 02:58:45.629639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624589 2564 flags.go:64] FLAG: --enable-server="true" Jul 08 02:58:45.629639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624592 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 08 02:58:45.629639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624605 2564 flags.go:64] FLAG: --event-burst="100" Jul 08 02:58:45.629639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624608 2564 flags.go:64] FLAG: --event-qps="50" Jul 08 02:58:45.629639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624612 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 08 02:58:45.629639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624615 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 08 02:58:45.629639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624619 2564 flags.go:64] FLAG: --eviction-hard="" Jul 08 02:58:45.629639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624623 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 08 02:58:45.629639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624625 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 08 02:58:45.629639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624628 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 08 02:58:45.629639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624631 2564 flags.go:64] FLAG: --eviction-soft="" Jul 08 02:58:45.629639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624634 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 08 02:58:45.629639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624637 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 08 02:58:45.629639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624640 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 08 02:58:45.630263 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624643 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 08 02:58:45.630263 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624646 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 08 02:58:45.630263 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624649 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 08 02:58:45.630263 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624651 2564 flags.go:64] FLAG: --feature-gates="" Jul 08 02:58:45.630263 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624655 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 08 02:58:45.630263 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624658 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 08 02:58:45.630263 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624661 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 08 02:58:45.630263 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624666 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 08 02:58:45.630263 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624669 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 08 02:58:45.630263 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624672 2564 flags.go:64] FLAG: --help="false" Jul 08 02:58:45.630263 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624674 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-128-70.ec2.internal" Jul 08 02:58:45.630263 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624677 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 08 02:58:45.630263 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624680 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 08 02:58:45.630263 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624683 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 08 02:58:45.630263 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624687 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 08 02:58:45.630263 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624690 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 08 02:58:45.630263 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624693 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 08 02:58:45.630263 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624696 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 08 02:58:45.630263 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624699 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 08 02:58:45.630263 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624701 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 08 02:58:45.630263 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624705 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 08 02:58:45.630263 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624708 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 08 02:58:45.630263 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624711 2564 flags.go:64] FLAG: --kube-reserved="" Jul 08 02:58:45.630263 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624714 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 08 02:58:45.630846 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624718 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 08 02:58:45.630846 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624721 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 08 02:58:45.630846 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624724 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 08 02:58:45.630846 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624727 2564 flags.go:64] FLAG: --lock-file="" Jul 08 02:58:45.630846 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624729 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 08 02:58:45.630846 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624732 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 08 02:58:45.630846 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624735 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 08 02:58:45.630846 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624740 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 08 02:58:45.630846 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624743 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 08 02:58:45.630846 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624745 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 08 02:58:45.630846 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624748 2564 flags.go:64] FLAG: --logging-format="text" Jul 08 02:58:45.630846 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624751 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 08 02:58:45.630846 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624754 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 08 02:58:45.630846 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624757 2564 flags.go:64] FLAG: --manifest-url="" Jul 08 02:58:45.630846 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624760 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 08 02:58:45.630846 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624765 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 08 02:58:45.630846 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624768 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 08 02:58:45.630846 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624772 2564 flags.go:64] FLAG: --max-pods="110" Jul 08 02:58:45.630846 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624775 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 08 02:58:45.630846 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624778 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 08 02:58:45.630846 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624781 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 08 02:58:45.630846 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624784 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 08 02:58:45.630846 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624787 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 08 02:58:45.630846 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624789 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 08 02:58:45.630846 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624792 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 08 02:58:45.631506 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624799 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 08 02:58:45.631506 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624802 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 08 02:58:45.631506 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624805 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 08 02:58:45.631506 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624808 2564 flags.go:64] FLAG: --pod-cidr="" Jul 08 02:58:45.631506 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624812 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 08 02:58:45.631506 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624818 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 08 02:58:45.631506 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624821 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 08 02:58:45.631506 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624825 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 08 02:58:45.631506 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624828 2564 flags.go:64] FLAG: --port="10250" Jul 08 02:58:45.631506 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624831 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 08 02:58:45.631506 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624834 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09f9896bbfe1a135d" Jul 08 02:58:45.631506 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624837 2564 flags.go:64] FLAG: --qos-reserved="" Jul 08 02:58:45.631506 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624840 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 08 02:58:45.631506 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624843 2564 flags.go:64] FLAG: --register-node="true" Jul 08 02:58:45.631506 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624846 2564 flags.go:64] FLAG: --register-with-taints="" Jul 08 02:58:45.631506 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624849 2564 flags.go:64] FLAG: --registry-burst="10" Jul 08 02:58:45.631506 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624852 2564 flags.go:64] FLAG: --registry-qps="5" Jul 08 02:58:45.631506 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624855 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 08 02:58:45.631506 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624858 2564 flags.go:64] FLAG: --reserved-memory="" Jul 08 02:58:45.631506 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624862 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 08 02:58:45.631506 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624865 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 08 02:58:45.631506 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624868 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 08 02:58:45.631506 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624871 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 08 02:58:45.631506 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624875 2564 flags.go:64] FLAG: --runonce="false" Jul 08 02:58:45.631506 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624878 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 08 02:58:45.632130 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624881 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 08 02:58:45.632130 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624883 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 08 02:58:45.632130 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624886 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 08 02:58:45.632130 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624889 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 08 02:58:45.632130 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624892 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 08 02:58:45.632130 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624895 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 08 02:58:45.632130 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624898 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 08 02:58:45.632130 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624901 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 08 02:58:45.632130 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624903 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 08 02:58:45.632130 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624906 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 08 02:58:45.632130 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624909 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 08 02:58:45.632130 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624912 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 08 02:58:45.632130 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624930 2564 flags.go:64] FLAG: --system-cgroups="" Jul 08 02:58:45.632130 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624933 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 08 02:58:45.632130 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624940 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 08 02:58:45.632130 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624943 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 08 02:58:45.632130 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624945 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 08 02:58:45.632130 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624950 2564 flags.go:64] FLAG: --tls-min-version="" Jul 08 02:58:45.632130 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624953 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 08 02:58:45.632130 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624956 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 08 02:58:45.632130 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624958 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 08 02:58:45.632130 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624961 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 08 02:58:45.632130 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624964 2564 flags.go:64] FLAG: --v="2" Jul 08 02:58:45.632130 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624968 2564 flags.go:64] FLAG: --version="false" Jul 08 02:58:45.632130 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624972 2564 flags.go:64] FLAG: --vmodule="" Jul 08 02:58:45.632732 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624975 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 08 02:58:45.632732 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.624978 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 08 02:58:45.632732 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625098 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 02:58:45.632732 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625101 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 02:58:45.632732 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625105 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 02:58:45.632732 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625109 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 02:58:45.632732 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625111 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 02:58:45.632732 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625114 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 02:58:45.632732 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625116 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 02:58:45.632732 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625119 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 02:58:45.632732 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625122 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 02:58:45.632732 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625124 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 02:58:45.632732 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625127 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 02:58:45.632732 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625129 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 02:58:45.632732 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625133 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 02:58:45.632732 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625137 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 02:58:45.632732 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625140 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 02:58:45.632732 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625143 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 02:58:45.632732 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625146 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 02:58:45.633199 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625149 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 02:58:45.633199 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625152 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 02:58:45.633199 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625156 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 02:58:45.633199 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625159 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 02:58:45.633199 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625161 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 02:58:45.633199 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625164 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 02:58:45.633199 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625166 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 02:58:45.633199 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625169 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 02:58:45.633199 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625172 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 02:58:45.633199 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625175 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 02:58:45.633199 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625177 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 02:58:45.633199 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625180 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 02:58:45.633199 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625182 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 02:58:45.633199 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625184 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 02:58:45.633199 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625187 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 02:58:45.633199 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625189 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 02:58:45.633199 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625191 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 02:58:45.633199 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625194 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 02:58:45.633199 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625199 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 02:58:45.633199 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625201 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 02:58:45.633680 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625203 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 02:58:45.633680 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625206 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 02:58:45.633680 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625208 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 02:58:45.633680 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625212 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 02:58:45.633680 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625216 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 02:58:45.633680 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625218 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 02:58:45.633680 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625221 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 02:58:45.633680 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625223 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 02:58:45.633680 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625226 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 02:58:45.633680 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625229 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 02:58:45.633680 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625231 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 02:58:45.633680 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625233 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 02:58:45.633680 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625235 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 02:58:45.633680 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625238 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 02:58:45.633680 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625245 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 02:58:45.633680 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625248 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 02:58:45.633680 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625250 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 02:58:45.633680 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625252 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 02:58:45.633680 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625255 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 02:58:45.634202 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625259 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 02:58:45.634202 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625262 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 02:58:45.634202 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625264 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 02:58:45.634202 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625267 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 02:58:45.634202 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625270 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 02:58:45.634202 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625273 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 02:58:45.634202 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625276 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 02:58:45.634202 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625278 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 02:58:45.634202 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625281 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 02:58:45.634202 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625283 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 02:58:45.634202 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625286 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 02:58:45.634202 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625290 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 08 02:58:45.634202 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625292 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 02:58:45.634202 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625295 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 02:58:45.634202 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625297 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 02:58:45.634202 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625300 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 02:58:45.634202 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625302 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 02:58:45.634202 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625305 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 02:58:45.634202 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625307 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 02:58:45.634202 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625309 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 02:58:45.634684 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625312 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 02:58:45.634684 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625314 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 02:58:45.634684 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625317 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 02:58:45.634684 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625319 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 02:58:45.634684 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625322 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 02:58:45.634684 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625324 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 02:58:45.634684 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625327 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 02:58:45.634684 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625331 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 02:58:45.634684 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625334 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 02:58:45.634684 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625336 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 02:58:45.634684 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625339 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 02:58:45.634684 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625341 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 02:58:45.634684 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625344 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 02:58:45.634684 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625346 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 02:58:45.634684 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625349 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 02:58:45.634684 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625351 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 02:58:45.634684 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625354 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 02:58:45.634684 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625356 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 02:58:45.634684 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625359 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 02:58:45.635224 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625361 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 02:58:45.635224 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625363 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 02:58:45.635224 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625366 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 02:58:45.635224 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.625368 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 02:58:45.635224 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.625375 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 02:58:45.635224 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.632833 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 08 02:58:45.635224 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.632851 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 08 02:58:45.635224 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.633591 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 08 02:58:45.635224 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.633598 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 08 02:58:45.635224 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633655 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 02:58:45.635224 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633661 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 02:58:45.635224 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633664 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 02:58:45.635224 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633668 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 02:58:45.635224 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633671 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 02:58:45.635224 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633674 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 02:58:45.635599 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633677 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 02:58:45.635599 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633680 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 02:58:45.635599 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633683 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 02:58:45.635599 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633686 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 02:58:45.635599 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633689 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 02:58:45.635599 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633691 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 02:58:45.635599 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633694 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 02:58:45.635599 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633697 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 02:58:45.635599 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633700 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 02:58:45.635599 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633702 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 02:58:45.635599 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633705 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 02:58:45.635599 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633707 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 02:58:45.635599 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633710 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 02:58:45.635599 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633713 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 02:58:45.635599 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633715 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 02:58:45.635599 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633718 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 02:58:45.635599 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633720 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 02:58:45.635599 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633723 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 02:58:45.635599 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633726 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 02:58:45.635599 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633728 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 02:58:45.636084 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633731 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 02:58:45.636084 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633734 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 02:58:45.636084 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633736 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 02:58:45.636084 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633741 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 02:58:45.636084 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633743 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 02:58:45.636084 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633746 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 02:58:45.636084 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633748 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 02:58:45.636084 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633751 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 02:58:45.636084 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633753 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 02:58:45.636084 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633755 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 02:58:45.636084 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633758 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 02:58:45.636084 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633760 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 02:58:45.636084 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633762 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 02:58:45.636084 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633765 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 02:58:45.636084 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633767 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 02:58:45.636084 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633770 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 02:58:45.636084 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633773 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 02:58:45.636084 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633775 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 02:58:45.636084 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633777 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 02:58:45.636084 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633780 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 02:58:45.636611 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633782 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 02:58:45.636611 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633785 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 02:58:45.636611 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633787 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 02:58:45.636611 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633791 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 02:58:45.636611 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633795 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 02:58:45.636611 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633798 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 02:58:45.636611 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633801 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 02:58:45.636611 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633803 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 02:58:45.636611 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633805 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 02:58:45.636611 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633808 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 02:58:45.636611 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633812 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 02:58:45.636611 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633815 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 02:58:45.636611 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633818 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 08 02:58:45.636611 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633821 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 02:58:45.636611 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633824 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 02:58:45.636611 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633827 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 02:58:45.636611 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633831 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 02:58:45.636611 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633833 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 02:58:45.636611 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633836 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 02:58:45.637077 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633838 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 02:58:45.637077 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633841 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 02:58:45.637077 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633843 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 02:58:45.637077 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633847 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 02:58:45.637077 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633850 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 02:58:45.637077 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633853 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 02:58:45.637077 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633856 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 02:58:45.637077 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633858 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 02:58:45.637077 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633861 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 02:58:45.637077 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633863 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 02:58:45.637077 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633866 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 02:58:45.637077 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633868 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 02:58:45.637077 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633871 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 02:58:45.637077 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633874 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 02:58:45.637077 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633876 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 02:58:45.637077 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633878 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 02:58:45.637077 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633881 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 02:58:45.637077 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633883 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 02:58:45.637077 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633886 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 02:58:45.637077 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633889 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 02:58:45.637555 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633891 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 02:58:45.637555 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633894 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 02:58:45.637555 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633896 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 02:58:45.637555 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633899 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 02:58:45.637555 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633902 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 02:58:45.637555 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633904 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 02:58:45.637555 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633907 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 02:58:45.637555 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633910 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 02:58:45.637555 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633912 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 02:58:45.637555 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633915 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 02:58:45.637555 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633936 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 02:58:45.637555 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633939 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 02:58:45.637555 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633942 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 02:58:45.637555 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.633944 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 02:58:45.637893 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.633950 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 02:58:45.637893 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634071 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 02:58:45.637893 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634077 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 02:58:45.637893 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634080 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 02:58:45.637893 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634084 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 02:58:45.637893 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634086 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 02:58:45.637893 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634089 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 02:58:45.637893 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634092 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 02:58:45.637893 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634094 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 02:58:45.637893 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634097 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 02:58:45.637893 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634099 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 02:58:45.637893 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634102 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 02:58:45.637893 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634104 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 02:58:45.637893 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634108 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 02:58:45.638268 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634111 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 02:58:45.638268 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634114 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 02:58:45.638268 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634117 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 02:58:45.638268 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634120 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 02:58:45.638268 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634122 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 02:58:45.638268 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634125 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 02:58:45.638268 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634128 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 02:58:45.638268 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634130 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 02:58:45.638268 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634133 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 02:58:45.638268 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634135 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 02:58:45.638268 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634138 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 02:58:45.638268 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634140 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 02:58:45.638268 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634143 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 02:58:45.638268 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634145 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 02:58:45.638268 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634148 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 02:58:45.638268 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634151 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 02:58:45.638268 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634153 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 02:58:45.638268 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634156 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 02:58:45.638268 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634158 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 02:58:45.638268 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634161 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 02:58:45.638739 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634163 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 02:58:45.638739 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634166 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 02:58:45.638739 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634168 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 02:58:45.638739 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634170 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 02:58:45.638739 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634173 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 02:58:45.638739 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634175 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 02:58:45.638739 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634177 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 02:58:45.638739 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634180 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 02:58:45.638739 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634182 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 02:58:45.638739 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634184 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 02:58:45.638739 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634187 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 02:58:45.638739 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634189 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 02:58:45.638739 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634191 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 02:58:45.638739 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634194 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 08 02:58:45.638739 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634196 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 02:58:45.638739 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634199 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 02:58:45.638739 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634201 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 02:58:45.638739 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634204 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 02:58:45.638739 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634206 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 02:58:45.638739 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634209 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 02:58:45.639243 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634212 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 02:58:45.639243 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634214 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 02:58:45.639243 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634216 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 02:58:45.639243 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634219 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 02:58:45.639243 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634222 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 02:58:45.639243 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634224 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 02:58:45.639243 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634227 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 02:58:45.639243 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634229 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 02:58:45.639243 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634232 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 02:58:45.639243 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634234 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 02:58:45.639243 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634237 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 02:58:45.639243 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634239 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 02:58:45.639243 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634242 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 02:58:45.639243 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634244 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 02:58:45.639243 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634247 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 02:58:45.639243 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634249 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 02:58:45.639243 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634252 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 02:58:45.639243 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634254 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 02:58:45.639243 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634256 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 02:58:45.639243 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634259 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 02:58:45.639722 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634262 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 02:58:45.639722 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634264 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 02:58:45.639722 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634266 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 02:58:45.639722 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634269 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 02:58:45.639722 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634271 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 02:58:45.639722 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634273 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 02:58:45.639722 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634276 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 02:58:45.639722 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634278 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 02:58:45.639722 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634281 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 02:58:45.639722 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634283 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 02:58:45.639722 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634285 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 02:58:45.639722 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634288 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 02:58:45.639722 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634291 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 02:58:45.639722 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634295 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 02:58:45.639722 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634297 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 02:58:45.639722 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634299 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 02:58:45.639722 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634302 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 02:58:45.639722 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634305 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 02:58:45.639722 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634307 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 02:58:45.639722 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634310 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 02:58:45.640292 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634312 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 02:58:45.640292 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634315 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 02:58:45.640292 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634317 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 02:58:45.640292 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634320 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 02:58:45.640292 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634323 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 02:58:45.640292 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:45.634326 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 02:58:45.640292 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.634331 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 02:58:45.640292 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.634461 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 08 02:58:45.640292 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.636406 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 08 02:58:45.640292 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.637891 2564 server.go:1019] "Starting client certificate rotation" Jul 08 02:58:45.640292 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.637992 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 02:58:45.640292 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.638027 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 02:58:45.640700 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.640686 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 02:58:45.647102 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.647079 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 02:58:45.657742 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.657721 2564 log.go:25] "Validated CRI v1 runtime API" Jul 08 02:58:45.659835 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.659820 2564 log.go:25] "Validated CRI v1 image API" Jul 08 02:58:45.660639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.660624 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 08 02:58:45.663121 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.663099 2564 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9fb1dbec-b0e5-4000-ac69-015f64f92417:/dev/nvme0n1p4 d8f87a8a-fd08-4fb5-8e66-7ecc430c511c:/dev/nvme0n1p3] Jul 08 02:58:45.663173 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.663123 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 08 02:58:45.667581 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.667465 2564 manager.go:216] Machine: {Timestamp:2026-07-08 02:58:45.666986166 +0000 UTC m=+0.324362028 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101578 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c35f00eab9c9afe6e24a710b4322b SystemUUID:ec2c35f0-0eab-9c9a-fe6e-24a710b4322b BootID:8184b1c1-f22b-45b9-b076-3efc860aca4f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e0:8a:15:d9:87 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e0:8a:15:d9:87 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:05:2a:e1:90:42 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 08 02:58:45.667581 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.667571 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 08 02:58:45.667707 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.667645 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 08 02:58:45.669502 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.669484 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 02:58:45.670882 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.670853 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 08 02:58:45.671066 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.670887 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-70.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 08 02:58:45.671624 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.671611 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 08 02:58:45.671681 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.671631 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 08 02:58:45.671681 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.671643 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 02:58:45.672387 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.672375 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 02:58:45.673490 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.673478 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 08 02:58:45.676125 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.676110 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 08 02:58:45.676263 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.676251 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 08 02:58:45.676317 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.676292 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 08 02:58:45.676317 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.676307 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 08 02:58:45.676411 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.676328 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 08 02:58:45.676411 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.676342 2564 kubelet.go:398] "Adding apiserver pod source" Jul 08 02:58:45.676411 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.676358 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 08 02:58:45.676765 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.676751 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 02:58:45.676831 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.676776 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 02:58:45.678953 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.678901 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 08 02:58:45.680960 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.680943 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 08 02:58:45.681019 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.680975 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 08 02:58:45.682563 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.682549 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 08 02:58:45.682606 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.682569 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 08 02:58:45.682606 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.682576 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 08 02:58:45.682606 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.682581 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 08 02:58:45.682606 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.682590 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 08 02:58:45.682606 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.682596 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 08 02:58:45.682606 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.682603 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 08 02:58:45.682606 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.682609 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 08 02:58:45.682799 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.682617 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 08 02:58:45.682799 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.682622 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 08 02:58:45.682799 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.682631 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 08 02:58:45.682799 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.682636 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 08 02:58:45.684057 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.684041 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 08 02:58:45.684107 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.684063 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 08 02:58:45.684758 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:45.684723 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-70.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 08 02:58:45.684826 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:45.684723 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 08 02:58:45.687116 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.687100 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-70.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 08 02:58:45.687347 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.687334 2564 server.go:1268] "Started kubelet" Jul 08 02:58:45.687504 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.687416 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 08 02:58:45.687569 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.687553 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 08 02:58:45.690632 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.690601 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 08 02:58:45.690845 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.690825 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 08 02:58:45.691223 ip-10-0-128-70 systemd[1]: Started Kubernetes Kubelet. Jul 08 02:58:45.691428 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.691387 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n7j6x" Jul 08 02:58:45.695227 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.695208 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 08 02:58:45.695732 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:45.695708 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 08 02:58:45.698200 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.698180 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n7j6x" Jul 08 02:58:45.698631 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:45.697671 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-70.ec2.internal.18c0312de8c6108f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-70.ec2.internal,UID:ip-10-0-128-70.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-70.ec2.internal,},FirstTimestamp:2026-07-08 02:58:45.687300239 +0000 UTC m=+0.344676102,LastTimestamp:2026-07-08 02:58:45.687300239 +0000 UTC m=+0.344676102,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-70.ec2.internal,}" Jul 08 02:58:45.699557 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.699545 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 08 02:58:45.699557 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.699550 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 08 02:58:45.699686 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:45.699625 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-70.ec2.internal\" not found" Jul 08 02:58:45.700171 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.700087 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 08 02:58:45.700171 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.700087 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 08 02:58:45.700171 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.700126 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 08 02:58:45.700408 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.700393 2564 factory.go:55] Registering systemd factory Jul 08 02:58:45.700457 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.700412 2564 factory.go:223] Registration of the systemd container factory successfully Jul 08 02:58:45.700838 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.700798 2564 factory.go:153] Registering CRI-O factory Jul 08 02:58:45.700838 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.700819 2564 factory.go:223] Registration of the crio container factory successfully Jul 08 02:58:45.701054 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.700879 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 08 02:58:45.701054 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.700903 2564 factory.go:103] Registering Raw factory Jul 08 02:58:45.701054 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.700937 2564 manager.go:1196] Started watching for new ooms in manager Jul 08 02:58:45.701336 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.701263 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 08 02:58:45.701336 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.701274 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 08 02:58:45.701487 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.701404 2564 manager.go:349] Starting recovery of all containers Jul 08 02:58:45.702192 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:45.702168 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-70.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 08 02:58:45.702315 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:45.702279 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 08 02:58:45.713133 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.713059 2564 manager.go:354] Recovery completed Jul 08 02:58:45.717776 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.717758 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 02:58:45.720580 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.720558 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasSufficientMemory" Jul 08 02:58:45.720678 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.720606 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 02:58:45.720678 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.720617 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasSufficientPID" Jul 08 02:58:45.721214 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.721198 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 08 02:58:45.721214 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.721211 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 08 02:58:45.721324 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.721226 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 08 02:58:45.721324 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.721250 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 02:58:45.722089 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.722073 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasSufficientMemory" Jul 08 02:58:45.722150 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.722103 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 02:58:45.722150 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.722117 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasSufficientPID" Jul 08 02:58:45.722533 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.722519 2564 policy_none.go:49] "None policy: Start" Jul 08 02:58:45.722570 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.722538 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 08 02:58:45.722570 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.722549 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 08 02:58:45.723517 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.723506 2564 policy_none.go:47] "Start" Jul 08 02:58:45.767608 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.767592 2564 manager.go:337] "Starting Device Plugin manager" Jul 08 02:58:45.767685 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:45.767637 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 08 02:58:45.767685 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.767650 2564 server.go:87] "Starting device plugin registration server" Jul 08 02:58:45.767971 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.767958 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 08 02:58:45.768014 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.767974 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 08 02:58:45.768094 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.768073 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 08 02:58:45.768168 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.768157 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 08 02:58:45.768168 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.768166 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 08 02:58:45.769082 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:45.769062 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 08 02:58:45.769188 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:45.769113 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-70.ec2.internal\" not found" Jul 08 02:58:45.868546 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.868462 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 02:58:45.869600 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.869476 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasSufficientMemory" Jul 08 02:58:45.869600 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.869505 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 02:58:45.869600 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.869515 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasSufficientPID" Jul 08 02:58:45.869600 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.869546 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-70.ec2.internal" Jul 08 02:58:45.876299 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.876282 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-70.ec2.internal" Jul 08 02:58:45.876372 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:45.876307 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-70.ec2.internal\": node \"ip-10-0-128-70.ec2.internal\" not found" Jul 08 02:58:45.878524 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.878499 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 08 02:58:45.879822 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.879804 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 08 02:58:45.879903 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.879846 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 08 02:58:45.879903 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.879871 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 08 02:58:45.880055 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:45.879965 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 08 02:58:45.882591 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.882565 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 02:58:45.891596 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:45.891569 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-70.ec2.internal\" not found" Jul 08 02:58:45.981400 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.980789 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-70.ec2.internal"] Jul 08 02:58:45.981502 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.981441 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 02:58:45.982527 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.982511 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasSufficientMemory" Jul 08 02:58:45.982592 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.982541 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 02:58:45.982592 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.982550 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasSufficientPID" Jul 08 02:58:45.983685 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.983672 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 02:58:45.983879 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.983866 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal" Jul 08 02:58:45.983961 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.983950 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 02:58:45.984484 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.984469 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasSufficientMemory" Jul 08 02:58:45.984545 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.984497 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 02:58:45.984545 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.984507 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasSufficientPID" Jul 08 02:58:45.984617 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.984470 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasSufficientMemory" Jul 08 02:58:45.984617 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.984567 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 02:58:45.984617 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.984577 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasSufficientPID" Jul 08 02:58:45.985621 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.985604 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-70.ec2.internal" Jul 08 02:58:45.985660 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.985641 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 02:58:45.986355 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.986340 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasSufficientMemory" Jul 08 02:58:45.986422 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.986364 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 02:58:45.986422 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:45.986376 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeHasSufficientPID" Jul 08 02:58:45.992387 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:45.992371 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-70.ec2.internal\" not found" Jul 08 02:58:46.003336 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:46.003315 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/233d4ffcc88e32f0bc8c7cf42f125937-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal\" (UID: \"233d4ffcc88e32f0bc8c7cf42f125937\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal" Jul 08 02:58:46.003450 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:46.003344 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/233d4ffcc88e32f0bc8c7cf42f125937-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal\" (UID: \"233d4ffcc88e32f0bc8c7cf42f125937\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal" Jul 08 02:58:46.003450 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:46.003362 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b68f5c5921db369bc5547aa00eae2349-config\") pod \"kube-apiserver-proxy-ip-10-0-128-70.ec2.internal\" (UID: \"b68f5c5921db369bc5547aa00eae2349\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-70.ec2.internal" Jul 08 02:58:46.005535 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:46.005521 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-70.ec2.internal\" not found" node="ip-10-0-128-70.ec2.internal" Jul 08 02:58:46.010034 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:46.010017 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-70.ec2.internal\" not found" node="ip-10-0-128-70.ec2.internal" Jul 08 02:58:46.092761 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:46.092727 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-70.ec2.internal\" not found" Jul 08 02:58:46.103908 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:46.103879 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/233d4ffcc88e32f0bc8c7cf42f125937-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal\" (UID: \"233d4ffcc88e32f0bc8c7cf42f125937\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal" Jul 08 02:58:46.104044 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:46.103914 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/233d4ffcc88e32f0bc8c7cf42f125937-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal\" (UID: \"233d4ffcc88e32f0bc8c7cf42f125937\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal" Jul 08 02:58:46.104044 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:46.103961 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b68f5c5921db369bc5547aa00eae2349-config\") pod \"kube-apiserver-proxy-ip-10-0-128-70.ec2.internal\" (UID: \"b68f5c5921db369bc5547aa00eae2349\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-70.ec2.internal" Jul 08 02:58:46.104044 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:46.103977 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/233d4ffcc88e32f0bc8c7cf42f125937-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal\" (UID: \"233d4ffcc88e32f0bc8c7cf42f125937\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal" Jul 08 02:58:46.104044 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:46.104013 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b68f5c5921db369bc5547aa00eae2349-config\") pod \"kube-apiserver-proxy-ip-10-0-128-70.ec2.internal\" (UID: \"b68f5c5921db369bc5547aa00eae2349\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-70.ec2.internal" Jul 08 02:58:46.104191 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:46.104055 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/233d4ffcc88e32f0bc8c7cf42f125937-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal\" (UID: \"233d4ffcc88e32f0bc8c7cf42f125937\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal" Jul 08 02:58:46.192990 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:46.192958 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-70.ec2.internal\" not found" Jul 08 02:58:46.293527 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:46.293493 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-70.ec2.internal\" not found" Jul 08 02:58:46.308986 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:46.308960 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal" Jul 08 02:58:46.313057 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:46.312873 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-70.ec2.internal" Jul 08 02:58:46.393635 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:46.393593 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-70.ec2.internal\" not found" Jul 08 02:58:46.494231 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:46.494156 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-70.ec2.internal\" not found" Jul 08 02:58:46.594687 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:46.594658 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-70.ec2.internal\" not found" Jul 08 02:58:46.614955 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:46.614911 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 02:58:46.637496 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:46.637465 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 08 02:58:46.638155 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:46.637614 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 08 02:58:46.638155 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:46.637619 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 08 02:58:46.695237 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:46.695204 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-70.ec2.internal\" not found" Jul 08 02:58:46.703929 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:46.703881 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 02:53:45 +0000 UTC" deadline="2028-04-03 10:55:00.041426924 +0000 UTC" Jul 08 02:58:46.703929 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:46.703915 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15247h56m13.33751634s" Jul 08 02:58:46.795300 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:46.795267 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-70.ec2.internal\" not found" Jul 08 02:58:46.795854 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:46.795838 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 02:58:46.845062 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:46.845026 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb68f5c5921db369bc5547aa00eae2349.slice/crio-b9f70dd1e2b9ced4a042cec05b048ded6df6dc3c0ebeb86726f7cea369ab9c87 WatchSource:0}: Error finding container b9f70dd1e2b9ced4a042cec05b048ded6df6dc3c0ebeb86726f7cea369ab9c87: Status 404 returned error can't find the container with id b9f70dd1e2b9ced4a042cec05b048ded6df6dc3c0ebeb86726f7cea369ab9c87 Jul 08 02:58:46.845235 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:46.845221 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod233d4ffcc88e32f0bc8c7cf42f125937.slice/crio-f04698e37070e378c925c805a0366a9abd92fd4737ca716302c68e97c8426cbd WatchSource:0}: Error finding container f04698e37070e378c925c805a0366a9abd92fd4737ca716302c68e97c8426cbd: Status 404 returned error can't find the container with id f04698e37070e378c925c805a0366a9abd92fd4737ca716302c68e97c8426cbd Jul 08 02:58:46.851014 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:46.850988 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 02:58:46.851974 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:46.851959 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 02:58:46.883286 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:46.883058 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal" event={"ID":"233d4ffcc88e32f0bc8c7cf42f125937","Type":"ContainerStarted","Data":"f04698e37070e378c925c805a0366a9abd92fd4737ca716302c68e97c8426cbd"} Jul 08 02:58:46.884080 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:46.884056 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-70.ec2.internal" event={"ID":"b68f5c5921db369bc5547aa00eae2349","Type":"ContainerStarted","Data":"b9f70dd1e2b9ced4a042cec05b048ded6df6dc3c0ebeb86726f7cea369ab9c87"} Jul 08 02:58:46.900202 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:46.900177 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal" Jul 08 02:58:46.911750 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:46.911725 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 02:58:46.912819 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:46.912805 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-70.ec2.internal" Jul 08 02:58:46.921582 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:46.921561 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 02:58:47.677419 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.677382 2564 apiserver.go:52] "Watching apiserver" Jul 08 02:58:47.686845 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.686814 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 08 02:58:47.687199 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.687175 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-lsmbz","openshift-network-diagnostics/network-check-target-hldxf","openshift-ovn-kubernetes/ovnkube-node-lv2jb","kube-system/konnectivity-agent-4ccc2","kube-system/kube-apiserver-proxy-ip-10-0-128-70.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6","openshift-cluster-node-tuning-operator/tuned-9zm24","openshift-dns/node-resolver-8jg82","openshift-image-registry/node-ca-7v4gn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal","openshift-network-operator/iptables-alerter-b6kz7","kube-system/global-pull-secret-syncer-54p5l","openshift-multus/multus-additional-cni-plugins-lwl2s","openshift-multus/multus-b25jn"] Jul 08 02:58:47.688729 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.688708 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.689848 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.689820 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b6kz7" Jul 08 02:58:47.691191 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.691171 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.691658 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.691641 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 08 02:58:47.691740 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.691683 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-v22j4\"" Jul 08 02:58:47.691740 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.691698 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 08 02:58:47.692222 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.692185 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 08 02:58:47.692307 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.692256 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4ccc2" Jul 08 02:58:47.692487 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.692471 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h2vcg\"" Jul 08 02:58:47.693009 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.692995 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 08 02:58:47.693289 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.693272 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 08 02:58:47.693618 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.693604 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5bjbp\"" Jul 08 02:58:47.693754 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.693742 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 08 02:58:47.693983 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.693966 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 08 02:58:47.694259 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.694247 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.694324 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.694308 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 08 02:58:47.694488 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.694476 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 08 02:58:47.694977 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.694964 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 08 02:58:47.698031 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.695258 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 08 02:58:47.698031 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.695599 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 08 02:58:47.698031 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.695876 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-fms2t\"" Jul 08 02:58:47.698031 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.696105 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 08 02:58:47.698031 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.696415 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6" Jul 08 02:58:47.698031 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.697321 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7v4gn" Jul 08 02:58:47.698031 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.697487 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 08 02:58:47.698031 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.697696 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 08 02:58:47.698031 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.697726 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lwl2s" Jul 08 02:58:47.698031 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.697727 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6rhz5\"" Jul 08 02:58:47.698031 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.697944 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 08 02:58:47.698604 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.698115 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 08 02:58:47.698604 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.698210 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:58:47.698604 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:47.698280 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hldxf" podUID="36d7c073-6c29-456f-a540-34e61812ac39" Jul 08 02:58:47.698604 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.698460 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 08 02:58:47.699438 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.699423 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8jg82" Jul 08 02:58:47.700217 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.700201 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 08 02:58:47.700628 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.700608 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:58:47.700704 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:47.700686 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lsmbz" podUID="8f5283b1-208f-415c-819f-c3e479102863" Jul 08 02:58:47.700877 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.700855 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 08 02:58:47.700974 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.700901 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-84b7w\"" Jul 08 02:58:47.701585 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.701564 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 08 02:58:47.701667 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.701625 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 08 02:58:47.702722 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.702703 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 08 02:58:47.703030 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.703014 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 08 02:58:47.703324 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.703307 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 08 02:58:47.703536 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.703520 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9d7lj\"" Jul 08 02:58:47.703773 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.703757 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-54l6t\"" Jul 08 02:58:47.704004 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.703990 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 08 02:58:47.705157 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.704822 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 08 02:58:47.705157 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.704954 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 08 02:58:47.705157 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.704989 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lxsm9\"" Jul 08 02:58:47.705157 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.705053 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:58:47.705387 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:47.705153 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54p5l" podUID="a6e66819-f708-4e22-81d4-46adeef611e0" Jul 08 02:58:47.710266 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.710234 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4krlr\" (UniqueName: \"kubernetes.io/projected/97c25edf-e111-4f3c-933f-9e45c6926b2c-kube-api-access-4krlr\") pod \"node-ca-7v4gn\" (UID: \"97c25edf-e111-4f3c-933f-9e45c6926b2c\") " pod="openshift-image-registry/node-ca-7v4gn" Jul 08 02:58:47.710355 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.710272 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-log-socket\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.710355 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.710295 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97c25edf-e111-4f3c-933f-9e45c6926b2c-host\") pod \"node-ca-7v4gn\" (UID: \"97c25edf-e111-4f3c-933f-9e45c6926b2c\") " pod="openshift-image-registry/node-ca-7v4gn" Jul 08 02:58:47.710355 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.710314 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-lib-modules\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.710488 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.710354 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-node-log\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.710488 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.710405 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-host-cni-netd\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.710488 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.710427 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-host-run-netns\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.710488 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.710444 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-host-run-multus-certs\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.710488 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.710461 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fce35c6b-7b91-4a96-8f3f-4227c1ef6127-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lwl2s\" (UID: \"fce35c6b-7b91-4a96-8f3f-4227c1ef6127\") " pod="openshift-multus/multus-additional-cni-plugins-lwl2s" Jul 08 02:58:47.710488 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.710482 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-host\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.710685 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.710496 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-host-slash\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.710685 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.710540 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/97c25edf-e111-4f3c-933f-9e45c6926b2c-serviceca\") pod \"node-ca-7v4gn\" (UID: \"97c25edf-e111-4f3c-933f-9e45c6926b2c\") " pod="openshift-image-registry/node-ca-7v4gn" Jul 08 02:58:47.710685 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.710568 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-host-run-k8s-cni-cncf-io\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.710685 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.710596 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fce35c6b-7b91-4a96-8f3f-4227c1ef6127-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lwl2s\" (UID: \"fce35c6b-7b91-4a96-8f3f-4227c1ef6127\") " pod="openshift-multus/multus-additional-cni-plugins-lwl2s" Jul 08 02:58:47.710685 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.710625 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-systemd-units\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.710685 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.710639 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2d241eb6-7dbd-4ad7-86ef-6ba3fd7bf61c-hosts-file\") pod \"node-resolver-8jg82\" (UID: \"2d241eb6-7dbd-4ad7-86ef-6ba3fd7bf61c\") " pod="openshift-dns/node-resolver-8jg82" Jul 08 02:58:47.710685 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.710658 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-sys\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.710685 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.710680 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-run-systemd\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.711035 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.710705 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.711035 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.710733 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt62t\" (UniqueName: \"kubernetes.io/projected/3d770ab2-b9c7-463f-bdf8-1e6249e18ea4-kube-api-access-dt62t\") pod \"aws-ebs-csi-driver-node-l5sw6\" (UID: \"3d770ab2-b9c7-463f-bdf8-1e6249e18ea4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6" Jul 08 02:58:47.711035 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.710767 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtjfp\" (UniqueName: \"kubernetes.io/projected/4040905f-871b-470e-a30e-c52ec426b0de-kube-api-access-xtjfp\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.711035 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.710797 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d770ab2-b9c7-463f-bdf8-1e6249e18ea4-socket-dir\") pod \"aws-ebs-csi-driver-node-l5sw6\" (UID: \"3d770ab2-b9c7-463f-bdf8-1e6249e18ea4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6" Jul 08 02:58:47.711035 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.710820 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d770ab2-b9c7-463f-bdf8-1e6249e18ea4-sys-fs\") pod \"aws-ebs-csi-driver-node-l5sw6\" (UID: \"3d770ab2-b9c7-463f-bdf8-1e6249e18ea4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6" Jul 08 02:58:47.711035 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.710853 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-var-lib-kubelet\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.711035 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.710875 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b08cdccc-fc17-4ddf-9140-078331f5e052-tmp\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.711035 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.710895 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsf69\" (UniqueName: \"kubernetes.io/projected/b08cdccc-fc17-4ddf-9140-078331f5e052-kube-api-access-vsf69\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.711035 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.710957 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-etc-openvswitch\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.711035 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.710982 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4040905f-871b-470e-a30e-c52ec426b0de-env-overrides\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.711035 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711008 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-multus-cni-dir\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.711035 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711032 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab708362-5418-4355-936b-870a73469b42-host-slash\") pod \"iptables-alerter-b6kz7\" (UID: \"ab708362-5418-4355-936b-870a73469b42\") " pod="openshift-network-operator/iptables-alerter-b6kz7" Jul 08 02:58:47.711548 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711050 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7cwv\" (UniqueName: \"kubernetes.io/projected/2d241eb6-7dbd-4ad7-86ef-6ba3fd7bf61c-kube-api-access-t7cwv\") pod \"node-resolver-8jg82\" (UID: \"2d241eb6-7dbd-4ad7-86ef-6ba3fd7bf61c\") " pod="openshift-dns/node-resolver-8jg82" Jul 08 02:58:47.711548 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711062 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-etc-sysctl-conf\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.711548 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711075 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqfbh\" (UniqueName: \"kubernetes.io/projected/8f5283b1-208f-415c-819f-c3e479102863-kube-api-access-mqfbh\") pod \"network-metrics-daemon-lsmbz\" (UID: \"8f5283b1-208f-415c-819f-c3e479102863\") " pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:58:47.711548 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711093 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9rln\" (UniqueName: \"kubernetes.io/projected/570103f4-bd12-4dd7-813a-7d3a2028ff68-kube-api-access-v9rln\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.711548 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711116 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhn8g\" (UniqueName: \"kubernetes.io/projected/fce35c6b-7b91-4a96-8f3f-4227c1ef6127-kube-api-access-fhn8g\") pod \"multus-additional-cni-plugins-lwl2s\" (UID: \"fce35c6b-7b91-4a96-8f3f-4227c1ef6127\") " pod="openshift-multus/multus-additional-cni-plugins-lwl2s" Jul 08 02:58:47.711548 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711139 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d770ab2-b9c7-463f-bdf8-1e6249e18ea4-device-dir\") pod \"aws-ebs-csi-driver-node-l5sw6\" (UID: \"3d770ab2-b9c7-463f-bdf8-1e6249e18ea4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6" Jul 08 02:58:47.711548 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711160 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh64r\" (UniqueName: \"kubernetes.io/projected/36d7c073-6c29-456f-a540-34e61812ac39-kube-api-access-nh64r\") pod \"network-check-target-hldxf\" (UID: \"36d7c073-6c29-456f-a540-34e61812ac39\") " pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:58:47.711548 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711177 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-run\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.711548 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711198 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-host-run-netns\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.711548 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711227 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-run-openvswitch\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.711548 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711256 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4040905f-871b-470e-a30e-c52ec426b0de-ovnkube-config\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.711548 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711280 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f5283b1-208f-415c-819f-c3e479102863-metrics-certs\") pod \"network-metrics-daemon-lsmbz\" (UID: \"8f5283b1-208f-415c-819f-c3e479102863\") " pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:58:47.711548 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711292 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-etc-systemd\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.711548 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711308 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-run-ovn\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.711548 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711328 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6c992baa-d9c2-4241-9c52-9c930569f591-agent-certs\") pod \"konnectivity-agent-4ccc2\" (UID: \"6c992baa-d9c2-4241-9c52-9c930569f591\") " pod="kube-system/konnectivity-agent-4ccc2" Jul 08 02:58:47.711548 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711349 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6c992baa-d9c2-4241-9c52-9c930569f591-konnectivity-ca\") pod \"konnectivity-agent-4ccc2\" (UID: \"6c992baa-d9c2-4241-9c52-9c930569f591\") " pod="kube-system/konnectivity-agent-4ccc2" Jul 08 02:58:47.712240 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711374 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ab708362-5418-4355-936b-870a73469b42-iptables-alerter-script\") pod \"iptables-alerter-b6kz7\" (UID: \"ab708362-5418-4355-936b-870a73469b42\") " pod="openshift-network-operator/iptables-alerter-b6kz7" Jul 08 02:58:47.712240 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711394 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d770ab2-b9c7-463f-bdf8-1e6249e18ea4-etc-selinux\") pod \"aws-ebs-csi-driver-node-l5sw6\" (UID: \"3d770ab2-b9c7-463f-bdf8-1e6249e18ea4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6" Jul 08 02:58:47.712240 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711432 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-host-run-ovn-kubernetes\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.712240 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711464 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-system-cni-dir\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.712240 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711486 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/570103f4-bd12-4dd7-813a-7d3a2028ff68-multus-daemon-config\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.712240 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711507 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fce35c6b-7b91-4a96-8f3f-4227c1ef6127-cni-binary-copy\") pod \"multus-additional-cni-plugins-lwl2s\" (UID: \"fce35c6b-7b91-4a96-8f3f-4227c1ef6127\") " pod="openshift-multus/multus-additional-cni-plugins-lwl2s" Jul 08 02:58:47.712240 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711529 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-etc-sysctl-d\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.712240 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711564 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4040905f-871b-470e-a30e-c52ec426b0de-ovn-node-metrics-cert\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.712240 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711589 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4040905f-871b-470e-a30e-c52ec426b0de-ovnkube-script-lib\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.712240 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711612 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-cnibin\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.712240 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711640 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-host-var-lib-cni-multus\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.712240 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711660 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fce35c6b-7b91-4a96-8f3f-4227c1ef6127-cnibin\") pod \"multus-additional-cni-plugins-lwl2s\" (UID: \"fce35c6b-7b91-4a96-8f3f-4227c1ef6127\") " pod="openshift-multus/multus-additional-cni-plugins-lwl2s" Jul 08 02:58:47.712240 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711683 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnbvp\" (UniqueName: \"kubernetes.io/projected/ab708362-5418-4355-936b-870a73469b42-kube-api-access-xnbvp\") pod \"iptables-alerter-b6kz7\" (UID: \"ab708362-5418-4355-936b-870a73469b42\") " pod="openshift-network-operator/iptables-alerter-b6kz7" Jul 08 02:58:47.712240 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711702 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d770ab2-b9c7-463f-bdf8-1e6249e18ea4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l5sw6\" (UID: \"3d770ab2-b9c7-463f-bdf8-1e6249e18ea4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6" Jul 08 02:58:47.712240 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711717 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b08cdccc-fc17-4ddf-9140-078331f5e052-etc-tuned\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.712240 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711747 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-var-lib-openvswitch\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.712958 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711766 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-host-cni-bin\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.712958 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711783 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-os-release\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.712958 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.711972 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-host-var-lib-kubelet\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.712958 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.712002 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fce35c6b-7b91-4a96-8f3f-4227c1ef6127-system-cni-dir\") pod \"multus-additional-cni-plugins-lwl2s\" (UID: \"fce35c6b-7b91-4a96-8f3f-4227c1ef6127\") " pod="openshift-multus/multus-additional-cni-plugins-lwl2s" Jul 08 02:58:47.712958 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.712021 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d770ab2-b9c7-463f-bdf8-1e6249e18ea4-registration-dir\") pod \"aws-ebs-csi-driver-node-l5sw6\" (UID: \"3d770ab2-b9c7-463f-bdf8-1e6249e18ea4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6" Jul 08 02:58:47.712958 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.712043 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-etc-modprobe-d\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.712958 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.712062 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-host-kubelet\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.712958 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.712076 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-multus-socket-dir-parent\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.712958 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.712089 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d241eb6-7dbd-4ad7-86ef-6ba3fd7bf61c-tmp-dir\") pod \"node-resolver-8jg82\" (UID: \"2d241eb6-7dbd-4ad7-86ef-6ba3fd7bf61c\") " pod="openshift-dns/node-resolver-8jg82" Jul 08 02:58:47.712958 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.712101 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-etc-kubernetes\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.712958 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.712115 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-etc-sysconfig\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.712958 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.712127 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fce35c6b-7b91-4a96-8f3f-4227c1ef6127-os-release\") pod \"multus-additional-cni-plugins-lwl2s\" (UID: \"fce35c6b-7b91-4a96-8f3f-4227c1ef6127\") " pod="openshift-multus/multus-additional-cni-plugins-lwl2s" Jul 08 02:58:47.712958 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.712155 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/570103f4-bd12-4dd7-813a-7d3a2028ff68-cni-binary-copy\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.712958 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.712183 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-hostroot\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.712958 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.712204 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-etc-kubernetes\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.712958 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.712226 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fce35c6b-7b91-4a96-8f3f-4227c1ef6127-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lwl2s\" (UID: \"fce35c6b-7b91-4a96-8f3f-4227c1ef6127\") " pod="openshift-multus/multus-additional-cni-plugins-lwl2s" Jul 08 02:58:47.713791 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.712248 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-host-var-lib-cni-bin\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.713791 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.712268 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-multus-conf-dir\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.716597 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.716573 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 02:58:47.730593 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.730569 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tgj45" Jul 08 02:58:47.739550 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.739527 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tgj45" Jul 08 02:58:47.801390 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.801359 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 08 02:58:47.812544 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812520 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-host-var-lib-cni-multus\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.812544 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812546 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fce35c6b-7b91-4a96-8f3f-4227c1ef6127-cnibin\") pod \"multus-additional-cni-plugins-lwl2s\" (UID: \"fce35c6b-7b91-4a96-8f3f-4227c1ef6127\") " pod="openshift-multus/multus-additional-cni-plugins-lwl2s" Jul 08 02:58:47.812544 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812562 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnbvp\" (UniqueName: \"kubernetes.io/projected/ab708362-5418-4355-936b-870a73469b42-kube-api-access-xnbvp\") pod \"iptables-alerter-b6kz7\" (UID: \"ab708362-5418-4355-936b-870a73469b42\") " pod="openshift-network-operator/iptables-alerter-b6kz7" Jul 08 02:58:47.812807 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812578 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d770ab2-b9c7-463f-bdf8-1e6249e18ea4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l5sw6\" (UID: \"3d770ab2-b9c7-463f-bdf8-1e6249e18ea4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6" Jul 08 02:58:47.812807 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812599 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b08cdccc-fc17-4ddf-9140-078331f5e052-etc-tuned\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.812807 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812620 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-var-lib-openvswitch\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.812807 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812629 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fce35c6b-7b91-4a96-8f3f-4227c1ef6127-cnibin\") pod \"multus-additional-cni-plugins-lwl2s\" (UID: \"fce35c6b-7b91-4a96-8f3f-4227c1ef6127\") " pod="openshift-multus/multus-additional-cni-plugins-lwl2s" Jul 08 02:58:47.812807 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812641 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-host-cni-bin\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.812807 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812671 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-os-release\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.812807 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812680 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d770ab2-b9c7-463f-bdf8-1e6249e18ea4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l5sw6\" (UID: \"3d770ab2-b9c7-463f-bdf8-1e6249e18ea4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6" Jul 08 02:58:47.812807 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812697 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-host-var-lib-kubelet\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.812807 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812701 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-var-lib-openvswitch\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.812807 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812629 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-host-var-lib-cni-multus\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.812807 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812673 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-host-cni-bin\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.812807 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812720 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fce35c6b-7b91-4a96-8f3f-4227c1ef6127-system-cni-dir\") pod \"multus-additional-cni-plugins-lwl2s\" (UID: \"fce35c6b-7b91-4a96-8f3f-4227c1ef6127\") " pod="openshift-multus/multus-additional-cni-plugins-lwl2s" Jul 08 02:58:47.812807 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812750 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d770ab2-b9c7-463f-bdf8-1e6249e18ea4-registration-dir\") pod \"aws-ebs-csi-driver-node-l5sw6\" (UID: \"3d770ab2-b9c7-463f-bdf8-1e6249e18ea4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6" Jul 08 02:58:47.812807 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812754 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fce35c6b-7b91-4a96-8f3f-4227c1ef6127-system-cni-dir\") pod \"multus-additional-cni-plugins-lwl2s\" (UID: \"fce35c6b-7b91-4a96-8f3f-4227c1ef6127\") " pod="openshift-multus/multus-additional-cni-plugins-lwl2s" Jul 08 02:58:47.812807 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812771 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-etc-modprobe-d\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.812807 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812786 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-host-var-lib-kubelet\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.812807 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812793 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-host-kubelet\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.813550 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812817 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a6e66819-f708-4e22-81d4-46adeef611e0-dbus\") pod \"global-pull-secret-syncer-54p5l\" (UID: \"a6e66819-f708-4e22-81d4-46adeef611e0\") " pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:58:47.813550 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812832 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d770ab2-b9c7-463f-bdf8-1e6249e18ea4-registration-dir\") pod \"aws-ebs-csi-driver-node-l5sw6\" (UID: \"3d770ab2-b9c7-463f-bdf8-1e6249e18ea4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6" Jul 08 02:58:47.813550 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812839 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-multus-socket-dir-parent\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.813550 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812862 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d241eb6-7dbd-4ad7-86ef-6ba3fd7bf61c-tmp-dir\") pod \"node-resolver-8jg82\" (UID: \"2d241eb6-7dbd-4ad7-86ef-6ba3fd7bf61c\") " pod="openshift-dns/node-resolver-8jg82" Jul 08 02:58:47.813550 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812889 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-etc-kubernetes\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.813550 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812905 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-os-release\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.813550 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812908 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-etc-sysconfig\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.813550 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812938 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 08 02:58:47.813550 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812949 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-etc-modprobe-d\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.813550 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812968 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fce35c6b-7b91-4a96-8f3f-4227c1ef6127-os-release\") pod \"multus-additional-cni-plugins-lwl2s\" (UID: \"fce35c6b-7b91-4a96-8f3f-4227c1ef6127\") " pod="openshift-multus/multus-additional-cni-plugins-lwl2s" Jul 08 02:58:47.813550 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812970 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-etc-sysconfig\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.813550 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812996 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-host-kubelet\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.813550 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.812997 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/570103f4-bd12-4dd7-813a-7d3a2028ff68-cni-binary-copy\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.813550 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813013 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fce35c6b-7b91-4a96-8f3f-4227c1ef6127-os-release\") pod \"multus-additional-cni-plugins-lwl2s\" (UID: \"fce35c6b-7b91-4a96-8f3f-4227c1ef6127\") " pod="openshift-multus/multus-additional-cni-plugins-lwl2s" Jul 08 02:58:47.813550 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813051 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-multus-socket-dir-parent\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.813550 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813095 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-etc-kubernetes\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.813550 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813114 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-hostroot\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.813550 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813128 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-etc-kubernetes\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.814409 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813142 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fce35c6b-7b91-4a96-8f3f-4227c1ef6127-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lwl2s\" (UID: \"fce35c6b-7b91-4a96-8f3f-4227c1ef6127\") " pod="openshift-multus/multus-additional-cni-plugins-lwl2s" Jul 08 02:58:47.814409 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813157 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-host-var-lib-cni-bin\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.814409 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813176 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-multus-conf-dir\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.814409 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813199 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-multus-conf-dir\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.814409 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813221 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-etc-kubernetes\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.814409 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813260 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4krlr\" (UniqueName: \"kubernetes.io/projected/97c25edf-e111-4f3c-933f-9e45c6926b2c-kube-api-access-4krlr\") pod \"node-ca-7v4gn\" (UID: \"97c25edf-e111-4f3c-933f-9e45c6926b2c\") " pod="openshift-image-registry/node-ca-7v4gn" Jul 08 02:58:47.814409 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813281 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d241eb6-7dbd-4ad7-86ef-6ba3fd7bf61c-tmp-dir\") pod \"node-resolver-8jg82\" (UID: \"2d241eb6-7dbd-4ad7-86ef-6ba3fd7bf61c\") " pod="openshift-dns/node-resolver-8jg82" Jul 08 02:58:47.814409 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813288 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-log-socket\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.814409 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813318 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97c25edf-e111-4f3c-933f-9e45c6926b2c-host\") pod \"node-ca-7v4gn\" (UID: \"97c25edf-e111-4f3c-933f-9e45c6926b2c\") " pod="openshift-image-registry/node-ca-7v4gn" Jul 08 02:58:47.814409 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813364 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-lib-modules\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.814409 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813385 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-node-log\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.814409 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813406 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-host-cni-netd\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.814409 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813428 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-host-run-netns\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.814409 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813464 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-host-run-multus-certs\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.814409 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813541 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fce35c6b-7b91-4a96-8f3f-4227c1ef6127-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lwl2s\" (UID: \"fce35c6b-7b91-4a96-8f3f-4227c1ef6127\") " pod="openshift-multus/multus-additional-cni-plugins-lwl2s" Jul 08 02:58:47.814409 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813565 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-host\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.814409 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813586 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-host-slash\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.814409 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813600 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-hostroot\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.815198 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813613 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/97c25edf-e111-4f3c-933f-9e45c6926b2c-serviceca\") pod \"node-ca-7v4gn\" (UID: \"97c25edf-e111-4f3c-933f-9e45c6926b2c\") " pod="openshift-image-registry/node-ca-7v4gn" Jul 08 02:58:47.815198 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813660 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-host-run-k8s-cni-cncf-io\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.815198 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813692 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fce35c6b-7b91-4a96-8f3f-4227c1ef6127-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lwl2s\" (UID: \"fce35c6b-7b91-4a96-8f3f-4227c1ef6127\") " pod="openshift-multus/multus-additional-cni-plugins-lwl2s" Jul 08 02:58:47.815198 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813719 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-systemd-units\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.815198 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813748 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a6e66819-f708-4e22-81d4-46adeef611e0-kubelet-config\") pod \"global-pull-secret-syncer-54p5l\" (UID: \"a6e66819-f708-4e22-81d4-46adeef611e0\") " pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:58:47.815198 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813814 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2d241eb6-7dbd-4ad7-86ef-6ba3fd7bf61c-hosts-file\") pod \"node-resolver-8jg82\" (UID: \"2d241eb6-7dbd-4ad7-86ef-6ba3fd7bf61c\") " pod="openshift-dns/node-resolver-8jg82" Jul 08 02:58:47.815198 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813875 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-sys\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.815198 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813901 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-run-systemd\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.815198 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813942 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fce35c6b-7b91-4a96-8f3f-4227c1ef6127-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lwl2s\" (UID: \"fce35c6b-7b91-4a96-8f3f-4227c1ef6127\") " pod="openshift-multus/multus-additional-cni-plugins-lwl2s" Jul 08 02:58:47.815198 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813985 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-host-run-k8s-cni-cncf-io\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.815198 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813993 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.815198 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814019 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/570103f4-bd12-4dd7-813a-7d3a2028ff68-cni-binary-copy\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.815198 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814028 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-systemd-units\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.815198 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814080 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/97c25edf-e111-4f3c-933f-9e45c6926b2c-serviceca\") pod \"node-ca-7v4gn\" (UID: \"97c25edf-e111-4f3c-933f-9e45c6926b2c\") " pod="openshift-image-registry/node-ca-7v4gn" Jul 08 02:58:47.815198 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814090 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2d241eb6-7dbd-4ad7-86ef-6ba3fd7bf61c-hosts-file\") pod \"node-resolver-8jg82\" (UID: \"2d241eb6-7dbd-4ad7-86ef-6ba3fd7bf61c\") " pod="openshift-dns/node-resolver-8jg82" Jul 08 02:58:47.815198 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814102 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fce35c6b-7b91-4a96-8f3f-4227c1ef6127-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lwl2s\" (UID: \"fce35c6b-7b91-4a96-8f3f-4227c1ef6127\") " pod="openshift-multus/multus-additional-cni-plugins-lwl2s" Jul 08 02:58:47.815198 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.813946 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.815939 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814131 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt62t\" (UniqueName: \"kubernetes.io/projected/3d770ab2-b9c7-463f-bdf8-1e6249e18ea4-kube-api-access-dt62t\") pod \"aws-ebs-csi-driver-node-l5sw6\" (UID: \"3d770ab2-b9c7-463f-bdf8-1e6249e18ea4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6" Jul 08 02:58:47.815939 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814134 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-run-systemd\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.815939 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814146 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtjfp\" (UniqueName: \"kubernetes.io/projected/4040905f-871b-470e-a30e-c52ec426b0de-kube-api-access-xtjfp\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.815939 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814167 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97c25edf-e111-4f3c-933f-9e45c6926b2c-host\") pod \"node-ca-7v4gn\" (UID: \"97c25edf-e111-4f3c-933f-9e45c6926b2c\") " pod="openshift-image-registry/node-ca-7v4gn" Jul 08 02:58:47.815939 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814176 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d770ab2-b9c7-463f-bdf8-1e6249e18ea4-socket-dir\") pod \"aws-ebs-csi-driver-node-l5sw6\" (UID: \"3d770ab2-b9c7-463f-bdf8-1e6249e18ea4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6" Jul 08 02:58:47.815939 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814193 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d770ab2-b9c7-463f-bdf8-1e6249e18ea4-sys-fs\") pod \"aws-ebs-csi-driver-node-l5sw6\" (UID: \"3d770ab2-b9c7-463f-bdf8-1e6249e18ea4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6" Jul 08 02:58:47.815939 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814197 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-lib-modules\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.815939 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814210 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-var-lib-kubelet\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.815939 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814224 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b08cdccc-fc17-4ddf-9140-078331f5e052-tmp\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.815939 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814236 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsf69\" (UniqueName: \"kubernetes.io/projected/b08cdccc-fc17-4ddf-9140-078331f5e052-kube-api-access-vsf69\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.815939 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814237 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-node-log\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.815939 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814250 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-etc-openvswitch\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.815939 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814268 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4040905f-871b-470e-a30e-c52ec426b0de-env-overrides\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.815939 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814275 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-host-cni-netd\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.815939 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814283 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-multus-cni-dir\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.815939 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814297 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab708362-5418-4355-936b-870a73469b42-host-slash\") pod \"iptables-alerter-b6kz7\" (UID: \"ab708362-5418-4355-936b-870a73469b42\") " pod="openshift-network-operator/iptables-alerter-b6kz7" Jul 08 02:58:47.815939 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814311 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7cwv\" (UniqueName: \"kubernetes.io/projected/2d241eb6-7dbd-4ad7-86ef-6ba3fd7bf61c-kube-api-access-t7cwv\") pod \"node-resolver-8jg82\" (UID: \"2d241eb6-7dbd-4ad7-86ef-6ba3fd7bf61c\") " pod="openshift-dns/node-resolver-8jg82" Jul 08 02:58:47.816574 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814316 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-host-run-multus-certs\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.816574 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814323 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-etc-sysctl-conf\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.816574 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814337 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqfbh\" (UniqueName: \"kubernetes.io/projected/8f5283b1-208f-415c-819f-c3e479102863-kube-api-access-mqfbh\") pod \"network-metrics-daemon-lsmbz\" (UID: \"8f5283b1-208f-415c-819f-c3e479102863\") " pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:58:47.816574 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814352 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6e66819-f708-4e22-81d4-46adeef611e0-original-pull-secret\") pod \"global-pull-secret-syncer-54p5l\" (UID: \"a6e66819-f708-4e22-81d4-46adeef611e0\") " pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:58:47.816574 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814372 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9rln\" (UniqueName: \"kubernetes.io/projected/570103f4-bd12-4dd7-813a-7d3a2028ff68-kube-api-access-v9rln\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.816574 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814389 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhn8g\" (UniqueName: \"kubernetes.io/projected/fce35c6b-7b91-4a96-8f3f-4227c1ef6127-kube-api-access-fhn8g\") pod \"multus-additional-cni-plugins-lwl2s\" (UID: \"fce35c6b-7b91-4a96-8f3f-4227c1ef6127\") " pod="openshift-multus/multus-additional-cni-plugins-lwl2s" Jul 08 02:58:47.816574 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814404 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d770ab2-b9c7-463f-bdf8-1e6249e18ea4-device-dir\") pod \"aws-ebs-csi-driver-node-l5sw6\" (UID: \"3d770ab2-b9c7-463f-bdf8-1e6249e18ea4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6" Jul 08 02:58:47.816574 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814417 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh64r\" (UniqueName: \"kubernetes.io/projected/36d7c073-6c29-456f-a540-34e61812ac39-kube-api-access-nh64r\") pod \"network-check-target-hldxf\" (UID: \"36d7c073-6c29-456f-a540-34e61812ac39\") " pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:58:47.816574 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814430 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-run\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.816574 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814442 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-host-run-netns\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.816574 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814456 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-run-openvswitch\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.816574 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814468 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4040905f-871b-470e-a30e-c52ec426b0de-ovnkube-config\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.816574 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814480 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f5283b1-208f-415c-819f-c3e479102863-metrics-certs\") pod \"network-metrics-daemon-lsmbz\" (UID: \"8f5283b1-208f-415c-819f-c3e479102863\") " pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:58:47.816574 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814498 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-etc-systemd\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.816574 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814512 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-run-ovn\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.816574 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814526 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6c992baa-d9c2-4241-9c52-9c930569f591-agent-certs\") pod \"konnectivity-agent-4ccc2\" (UID: \"6c992baa-d9c2-4241-9c52-9c930569f591\") " pod="kube-system/konnectivity-agent-4ccc2" Jul 08 02:58:47.816574 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814532 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-sys\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.817177 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814538 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6c992baa-d9c2-4241-9c52-9c930569f591-konnectivity-ca\") pod \"konnectivity-agent-4ccc2\" (UID: \"6c992baa-d9c2-4241-9c52-9c930569f591\") " pod="kube-system/konnectivity-agent-4ccc2" Jul 08 02:58:47.817177 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814572 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ab708362-5418-4355-936b-870a73469b42-iptables-alerter-script\") pod \"iptables-alerter-b6kz7\" (UID: \"ab708362-5418-4355-936b-870a73469b42\") " pod="openshift-network-operator/iptables-alerter-b6kz7" Jul 08 02:58:47.817177 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814597 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d770ab2-b9c7-463f-bdf8-1e6249e18ea4-etc-selinux\") pod \"aws-ebs-csi-driver-node-l5sw6\" (UID: \"3d770ab2-b9c7-463f-bdf8-1e6249e18ea4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6" Jul 08 02:58:47.817177 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814620 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-host-run-ovn-kubernetes\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.817177 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814652 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-system-cni-dir\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.817177 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814674 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/570103f4-bd12-4dd7-813a-7d3a2028ff68-multus-daemon-config\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.817177 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814696 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fce35c6b-7b91-4a96-8f3f-4227c1ef6127-cni-binary-copy\") pod \"multus-additional-cni-plugins-lwl2s\" (UID: \"fce35c6b-7b91-4a96-8f3f-4227c1ef6127\") " pod="openshift-multus/multus-additional-cni-plugins-lwl2s" Jul 08 02:58:47.817177 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814713 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-host\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.817177 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814725 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fce35c6b-7b91-4a96-8f3f-4227c1ef6127-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lwl2s\" (UID: \"fce35c6b-7b91-4a96-8f3f-4227c1ef6127\") " pod="openshift-multus/multus-additional-cni-plugins-lwl2s" Jul 08 02:58:47.817177 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814735 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-host-slash\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.817177 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814772 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-var-lib-kubelet\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.817177 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814774 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-log-socket\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.817177 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814804 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d770ab2-b9c7-463f-bdf8-1e6249e18ea4-etc-selinux\") pod \"aws-ebs-csi-driver-node-l5sw6\" (UID: \"3d770ab2-b9c7-463f-bdf8-1e6249e18ea4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6" Jul 08 02:58:47.817177 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814810 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-etc-sysctl-d\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.817177 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814808 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-host-run-ovn-kubernetes\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.817177 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814856 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-system-cni-dir\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.817177 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814869 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d770ab2-b9c7-463f-bdf8-1e6249e18ea4-socket-dir\") pod \"aws-ebs-csi-driver-node-l5sw6\" (UID: \"3d770ab2-b9c7-463f-bdf8-1e6249e18ea4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6" Jul 08 02:58:47.817790 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814943 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d770ab2-b9c7-463f-bdf8-1e6249e18ea4-sys-fs\") pod \"aws-ebs-csi-driver-node-l5sw6\" (UID: \"3d770ab2-b9c7-463f-bdf8-1e6249e18ea4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6" Jul 08 02:58:47.817790 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814975 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6c992baa-d9c2-4241-9c52-9c930569f591-konnectivity-ca\") pod \"konnectivity-agent-4ccc2\" (UID: \"6c992baa-d9c2-4241-9c52-9c930569f591\") " pod="kube-system/konnectivity-agent-4ccc2" Jul 08 02:58:47.817790 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.814717 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-etc-sysctl-d\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.817790 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.815068 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4040905f-871b-470e-a30e-c52ec426b0de-ovn-node-metrics-cert\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.817790 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.815111 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4040905f-871b-470e-a30e-c52ec426b0de-ovnkube-script-lib\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.817790 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.815133 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-cnibin\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.817790 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.815136 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-host-var-lib-cni-bin\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.817790 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.815173 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-run\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.817790 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.815222 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-etc-systemd\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.817790 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:47.815230 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:58:47.817790 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.815240 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-etc-openvswitch\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.817790 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.815267 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-run-ovn\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.817790 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:47.815289 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f5283b1-208f-415c-819f-c3e479102863-metrics-certs podName:8f5283b1-208f-415c-819f-c3e479102863 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:48.315270353 +0000 UTC m=+2.972646214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f5283b1-208f-415c-819f-c3e479102863-metrics-certs") pod "network-metrics-daemon-lsmbz" (UID: "8f5283b1-208f-415c-819f-c3e479102863") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:58:47.817790 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.815316 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/570103f4-bd12-4dd7-813a-7d3a2028ff68-multus-daemon-config\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.817790 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.815324 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-host-run-netns\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.817790 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.815316 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fce35c6b-7b91-4a96-8f3f-4227c1ef6127-cni-binary-copy\") pod \"multus-additional-cni-plugins-lwl2s\" (UID: \"fce35c6b-7b91-4a96-8f3f-4227c1ef6127\") " pod="openshift-multus/multus-additional-cni-plugins-lwl2s" Jul 08 02:58:47.817790 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.815358 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4040905f-871b-470e-a30e-c52ec426b0de-run-openvswitch\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.818499 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.815387 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-host-run-netns\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.818499 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.815429 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab708362-5418-4355-936b-870a73469b42-host-slash\") pod \"iptables-alerter-b6kz7\" (UID: \"ab708362-5418-4355-936b-870a73469b42\") " pod="openshift-network-operator/iptables-alerter-b6kz7" Jul 08 02:58:47.818499 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.815642 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ab708362-5418-4355-936b-870a73469b42-iptables-alerter-script\") pod \"iptables-alerter-b6kz7\" (UID: \"ab708362-5418-4355-936b-870a73469b42\") " pod="openshift-network-operator/iptables-alerter-b6kz7" Jul 08 02:58:47.818499 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.815706 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-cnibin\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.818499 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.815851 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d770ab2-b9c7-463f-bdf8-1e6249e18ea4-device-dir\") pod \"aws-ebs-csi-driver-node-l5sw6\" (UID: \"3d770ab2-b9c7-463f-bdf8-1e6249e18ea4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6" Jul 08 02:58:47.818499 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.815874 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4040905f-871b-470e-a30e-c52ec426b0de-ovnkube-config\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.818499 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.815967 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/570103f4-bd12-4dd7-813a-7d3a2028ff68-multus-cni-dir\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.818499 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.815981 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b08cdccc-fc17-4ddf-9140-078331f5e052-etc-sysctl-conf\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.818499 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.816082 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4040905f-871b-470e-a30e-c52ec426b0de-ovnkube-script-lib\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.818499 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.816440 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4040905f-871b-470e-a30e-c52ec426b0de-env-overrides\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.818499 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.817345 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b08cdccc-fc17-4ddf-9140-078331f5e052-etc-tuned\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.818499 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.817514 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b08cdccc-fc17-4ddf-9140-078331f5e052-tmp\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.818499 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.818369 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4040905f-871b-470e-a30e-c52ec426b0de-ovn-node-metrics-cert\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.818499 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.818429 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6c992baa-d9c2-4241-9c52-9c930569f591-agent-certs\") pod \"konnectivity-agent-4ccc2\" (UID: \"6c992baa-d9c2-4241-9c52-9c930569f591\") " pod="kube-system/konnectivity-agent-4ccc2" Jul 08 02:58:47.822354 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.822332 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnbvp\" (UniqueName: \"kubernetes.io/projected/ab708362-5418-4355-936b-870a73469b42-kube-api-access-xnbvp\") pod \"iptables-alerter-b6kz7\" (UID: \"ab708362-5418-4355-936b-870a73469b42\") " pod="openshift-network-operator/iptables-alerter-b6kz7" Jul 08 02:58:47.823049 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.823025 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsf69\" (UniqueName: \"kubernetes.io/projected/b08cdccc-fc17-4ddf-9140-078331f5e052-kube-api-access-vsf69\") pod \"tuned-9zm24\" (UID: \"b08cdccc-fc17-4ddf-9140-078331f5e052\") " pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:47.823161 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:47.823143 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 02:58:47.823238 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:47.823166 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 02:58:47.823238 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:47.823180 2564 projected.go:196] Error preparing data for projected volume kube-api-access-nh64r for pod openshift-network-diagnostics/network-check-target-hldxf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:58:47.823336 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:47.823256 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36d7c073-6c29-456f-a540-34e61812ac39-kube-api-access-nh64r podName:36d7c073-6c29-456f-a540-34e61812ac39 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:48.323232754 +0000 UTC m=+2.980608616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nh64r" (UniqueName: "kubernetes.io/projected/36d7c073-6c29-456f-a540-34e61812ac39-kube-api-access-nh64r") pod "network-check-target-hldxf" (UID: "36d7c073-6c29-456f-a540-34e61812ac39") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:58:47.823400 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.823358 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4krlr\" (UniqueName: \"kubernetes.io/projected/97c25edf-e111-4f3c-933f-9e45c6926b2c-kube-api-access-4krlr\") pod \"node-ca-7v4gn\" (UID: \"97c25edf-e111-4f3c-933f-9e45c6926b2c\") " pod="openshift-image-registry/node-ca-7v4gn" Jul 08 02:58:47.824329 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.824301 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9rln\" (UniqueName: \"kubernetes.io/projected/570103f4-bd12-4dd7-813a-7d3a2028ff68-kube-api-access-v9rln\") pod \"multus-b25jn\" (UID: \"570103f4-bd12-4dd7-813a-7d3a2028ff68\") " pod="openshift-multus/multus-b25jn" Jul 08 02:58:47.825376 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.825358 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtjfp\" (UniqueName: \"kubernetes.io/projected/4040905f-871b-470e-a30e-c52ec426b0de-kube-api-access-xtjfp\") pod \"ovnkube-node-lv2jb\" (UID: \"4040905f-871b-470e-a30e-c52ec426b0de\") " pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:47.825711 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.825689 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhn8g\" (UniqueName: \"kubernetes.io/projected/fce35c6b-7b91-4a96-8f3f-4227c1ef6127-kube-api-access-fhn8g\") pod \"multus-additional-cni-plugins-lwl2s\" (UID: \"fce35c6b-7b91-4a96-8f3f-4227c1ef6127\") " pod="openshift-multus/multus-additional-cni-plugins-lwl2s" Jul 08 02:58:47.826102 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.826067 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt62t\" (UniqueName: \"kubernetes.io/projected/3d770ab2-b9c7-463f-bdf8-1e6249e18ea4-kube-api-access-dt62t\") pod \"aws-ebs-csi-driver-node-l5sw6\" (UID: \"3d770ab2-b9c7-463f-bdf8-1e6249e18ea4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6" Jul 08 02:58:47.826779 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.826760 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqfbh\" (UniqueName: \"kubernetes.io/projected/8f5283b1-208f-415c-819f-c3e479102863-kube-api-access-mqfbh\") pod \"network-metrics-daemon-lsmbz\" (UID: \"8f5283b1-208f-415c-819f-c3e479102863\") " pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:58:47.827305 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.827284 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7cwv\" (UniqueName: \"kubernetes.io/projected/2d241eb6-7dbd-4ad7-86ef-6ba3fd7bf61c-kube-api-access-t7cwv\") pod \"node-resolver-8jg82\" (UID: \"2d241eb6-7dbd-4ad7-86ef-6ba3fd7bf61c\") " pod="openshift-dns/node-resolver-8jg82" Jul 08 02:58:47.916144 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.916112 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a6e66819-f708-4e22-81d4-46adeef611e0-kubelet-config\") pod \"global-pull-secret-syncer-54p5l\" (UID: \"a6e66819-f708-4e22-81d4-46adeef611e0\") " pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:58:47.916303 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.916163 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6e66819-f708-4e22-81d4-46adeef611e0-original-pull-secret\") pod \"global-pull-secret-syncer-54p5l\" (UID: \"a6e66819-f708-4e22-81d4-46adeef611e0\") " pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:58:47.916303 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.916258 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a6e66819-f708-4e22-81d4-46adeef611e0-kubelet-config\") pod \"global-pull-secret-syncer-54p5l\" (UID: \"a6e66819-f708-4e22-81d4-46adeef611e0\") " pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:58:47.916402 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.916312 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a6e66819-f708-4e22-81d4-46adeef611e0-dbus\") pod \"global-pull-secret-syncer-54p5l\" (UID: \"a6e66819-f708-4e22-81d4-46adeef611e0\") " pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:58:47.916402 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:47.916351 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 02:58:47.916502 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:47.916411 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e66819-f708-4e22-81d4-46adeef611e0-original-pull-secret podName:a6e66819-f708-4e22-81d4-46adeef611e0 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:48.416393584 +0000 UTC m=+3.073769444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6e66819-f708-4e22-81d4-46adeef611e0-original-pull-secret") pod "global-pull-secret-syncer-54p5l" (UID: "a6e66819-f708-4e22-81d4-46adeef611e0") : object "kube-system"/"original-pull-secret" not registered Jul 08 02:58:47.916502 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:47.916452 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a6e66819-f708-4e22-81d4-46adeef611e0-dbus\") pod \"global-pull-secret-syncer-54p5l\" (UID: \"a6e66819-f708-4e22-81d4-46adeef611e0\") " pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:58:48.001353 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.001238 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9zm24" Jul 08 02:58:48.008995 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.008971 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b6kz7" Jul 08 02:58:48.017363 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.017340 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:58:48.024024 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.023995 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4ccc2" Jul 08 02:58:48.029888 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.029871 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b25jn" Jul 08 02:58:48.038046 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.038023 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6" Jul 08 02:58:48.049531 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.049509 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7v4gn" Jul 08 02:58:48.058665 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.058645 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lwl2s" Jul 08 02:58:48.062878 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.062861 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8jg82" Jul 08 02:58:48.147177 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.147148 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 02:58:48.320665 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.320604 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f5283b1-208f-415c-819f-c3e479102863-metrics-certs\") pod \"network-metrics-daemon-lsmbz\" (UID: \"8f5283b1-208f-415c-819f-c3e479102863\") " pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:58:48.320791 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:48.320702 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:58:48.320791 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:48.320746 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f5283b1-208f-415c-819f-c3e479102863-metrics-certs podName:8f5283b1-208f-415c-819f-c3e479102863 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:49.320735125 +0000 UTC m=+3.978110972 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f5283b1-208f-415c-819f-c3e479102863-metrics-certs") pod "network-metrics-daemon-lsmbz" (UID: "8f5283b1-208f-415c-819f-c3e479102863") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:58:48.344843 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.344820 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 02:58:48.421050 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.421018 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6e66819-f708-4e22-81d4-46adeef611e0-original-pull-secret\") pod \"global-pull-secret-syncer-54p5l\" (UID: \"a6e66819-f708-4e22-81d4-46adeef611e0\") " pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:58:48.421209 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.421065 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh64r\" (UniqueName: \"kubernetes.io/projected/36d7c073-6c29-456f-a540-34e61812ac39-kube-api-access-nh64r\") pod \"network-check-target-hldxf\" (UID: \"36d7c073-6c29-456f-a540-34e61812ac39\") " pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:58:48.421209 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:48.421149 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 02:58:48.421209 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:48.421182 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 02:58:48.421209 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:48.421197 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 02:58:48.421209 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:48.421199 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e66819-f708-4e22-81d4-46adeef611e0-original-pull-secret podName:a6e66819-f708-4e22-81d4-46adeef611e0 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:49.421186259 +0000 UTC m=+4.078562109 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6e66819-f708-4e22-81d4-46adeef611e0-original-pull-secret") pod "global-pull-secret-syncer-54p5l" (UID: "a6e66819-f708-4e22-81d4-46adeef611e0") : object "kube-system"/"original-pull-secret" not registered Jul 08 02:58:48.421209 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:48.421204 2564 projected.go:196] Error preparing data for projected volume kube-api-access-nh64r for pod openshift-network-diagnostics/network-check-target-hldxf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:58:48.421413 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:48.421239 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36d7c073-6c29-456f-a540-34e61812ac39-kube-api-access-nh64r podName:36d7c073-6c29-456f-a540-34e61812ac39 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:49.421228635 +0000 UTC m=+4.078604486 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nh64r" (UniqueName: "kubernetes.io/projected/36d7c073-6c29-456f-a540-34e61812ac39-kube-api-access-nh64r") pod "network-check-target-hldxf" (UID: "36d7c073-6c29-456f-a540-34e61812ac39") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:58:48.639017 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:48.638987 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4040905f_871b_470e_a30e_c52ec426b0de.slice/crio-6e02703e7bfd3579d12f5a14b834c2cf45c22605fc5c793733d50a76623d3e40 WatchSource:0}: Error finding container 6e02703e7bfd3579d12f5a14b834c2cf45c22605fc5c793733d50a76623d3e40: Status 404 returned error can't find the container with id 6e02703e7bfd3579d12f5a14b834c2cf45c22605fc5c793733d50a76623d3e40 Jul 08 02:58:48.639845 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:48.639814 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08cdccc_fc17_4ddf_9140_078331f5e052.slice/crio-4aa98d8d6e61bdd89c2228a66132cbc9018a6b2661cd12d143ad14f741c96557 WatchSource:0}: Error finding container 4aa98d8d6e61bdd89c2228a66132cbc9018a6b2661cd12d143ad14f741c96557: Status 404 returned error can't find the container with id 4aa98d8d6e61bdd89c2228a66132cbc9018a6b2661cd12d143ad14f741c96557 Jul 08 02:58:48.641643 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:48.641076 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab708362_5418_4355_936b_870a73469b42.slice/crio-a5feb05a37d64e68b1e9a90e6f7c770d0309800f6aff7a6b94e12ad04d4b7095 WatchSource:0}: Error finding container a5feb05a37d64e68b1e9a90e6f7c770d0309800f6aff7a6b94e12ad04d4b7095: Status 404 returned error can't find the container with id a5feb05a37d64e68b1e9a90e6f7c770d0309800f6aff7a6b94e12ad04d4b7095 Jul 08 02:58:48.643492 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:48.643473 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c25edf_e111_4f3c_933f_9e45c6926b2c.slice/crio-c642117abe59fbd148fb0c76d0b17c4d68bf9235279a3c3d6584af144cd7790e WatchSource:0}: Error finding container c642117abe59fbd148fb0c76d0b17c4d68bf9235279a3c3d6584af144cd7790e: Status 404 returned error can't find the container with id c642117abe59fbd148fb0c76d0b17c4d68bf9235279a3c3d6584af144cd7790e Jul 08 02:58:48.644230 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:48.644210 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d770ab2_b9c7_463f_bdf8_1e6249e18ea4.slice/crio-f6c1ad64795bc13f500b0babbc24c2739a99acd0508b1b0292a088f7b50c1c22 WatchSource:0}: Error finding container f6c1ad64795bc13f500b0babbc24c2739a99acd0508b1b0292a088f7b50c1c22: Status 404 returned error can't find the container with id f6c1ad64795bc13f500b0babbc24c2739a99acd0508b1b0292a088f7b50c1c22 Jul 08 02:58:48.648640 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:48.648619 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d241eb6_7dbd_4ad7_86ef_6ba3fd7bf61c.slice/crio-599642a32fdd60232d19c8166b893243c2f364acba62dcef65ae3f4fc1c7160c WatchSource:0}: Error finding container 599642a32fdd60232d19c8166b893243c2f364acba62dcef65ae3f4fc1c7160c: Status 404 returned error can't find the container with id 599642a32fdd60232d19c8166b893243c2f364acba62dcef65ae3f4fc1c7160c Jul 08 02:58:48.649699 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:48.649677 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce35c6b_7b91_4a96_8f3f_4227c1ef6127.slice/crio-130442125c6ea4b29da8e293e87b7689ef7581039d11a14b3ab0685546a178b3 WatchSource:0}: Error finding container 130442125c6ea4b29da8e293e87b7689ef7581039d11a14b3ab0685546a178b3: Status 404 returned error can't find the container with id 130442125c6ea4b29da8e293e87b7689ef7581039d11a14b3ab0685546a178b3 Jul 08 02:58:48.650323 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:48.650305 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c992baa_d9c2_4241_9c52_9c930569f591.slice/crio-27282d9908376021bf8ba9e036daae6f4fcadb7527033503d2dcc7fbb09bfc15 WatchSource:0}: Error finding container 27282d9908376021bf8ba9e036daae6f4fcadb7527033503d2dcc7fbb09bfc15: Status 404 returned error can't find the container with id 27282d9908376021bf8ba9e036daae6f4fcadb7527033503d2dcc7fbb09bfc15 Jul 08 02:58:48.651720 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:58:48.651495 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod570103f4_bd12_4dd7_813a_7d3a2028ff68.slice/crio-661416509ec346b08acdf43913ab62aa2887fe0fd3debcd709eeba1b09d6406d WatchSource:0}: Error finding container 661416509ec346b08acdf43913ab62aa2887fe0fd3debcd709eeba1b09d6406d: Status 404 returned error can't find the container with id 661416509ec346b08acdf43913ab62aa2887fe0fd3debcd709eeba1b09d6406d Jul 08 02:58:48.742258 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.742212 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 02:53:47 +0000 UTC" deadline="2027-12-05 07:22:21.032888597 +0000 UTC" Jul 08 02:58:48.742258 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.742253 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12364h23m32.29064042s" Jul 08 02:58:48.880455 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.880429 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:58:48.880551 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:48.880524 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lsmbz" podUID="8f5283b1-208f-415c-819f-c3e479102863" Jul 08 02:58:48.889499 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.889464 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-70.ec2.internal" event={"ID":"b68f5c5921db369bc5547aa00eae2349","Type":"ContainerStarted","Data":"914bfcb48528e3d1e834899ac61da74bb515166720b32516d6d085b6789b8f5d"} Jul 08 02:58:48.890832 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.890803 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4ccc2" event={"ID":"6c992baa-d9c2-4241-9c52-9c930569f591","Type":"ContainerStarted","Data":"27282d9908376021bf8ba9e036daae6f4fcadb7527033503d2dcc7fbb09bfc15"} Jul 08 02:58:48.891948 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.891899 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwl2s" event={"ID":"fce35c6b-7b91-4a96-8f3f-4227c1ef6127","Type":"ContainerStarted","Data":"130442125c6ea4b29da8e293e87b7689ef7581039d11a14b3ab0685546a178b3"} Jul 08 02:58:48.892897 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.892876 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8jg82" event={"ID":"2d241eb6-7dbd-4ad7-86ef-6ba3fd7bf61c","Type":"ContainerStarted","Data":"599642a32fdd60232d19c8166b893243c2f364acba62dcef65ae3f4fc1c7160c"} Jul 08 02:58:48.893968 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.893946 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6" event={"ID":"3d770ab2-b9c7-463f-bdf8-1e6249e18ea4","Type":"ContainerStarted","Data":"f6c1ad64795bc13f500b0babbc24c2739a99acd0508b1b0292a088f7b50c1c22"} Jul 08 02:58:48.894945 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.894910 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7v4gn" event={"ID":"97c25edf-e111-4f3c-933f-9e45c6926b2c","Type":"ContainerStarted","Data":"c642117abe59fbd148fb0c76d0b17c4d68bf9235279a3c3d6584af144cd7790e"} Jul 08 02:58:48.895839 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.895819 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b6kz7" event={"ID":"ab708362-5418-4355-936b-870a73469b42","Type":"ContainerStarted","Data":"a5feb05a37d64e68b1e9a90e6f7c770d0309800f6aff7a6b94e12ad04d4b7095"} Jul 08 02:58:48.896765 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.896747 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b25jn" event={"ID":"570103f4-bd12-4dd7-813a-7d3a2028ff68","Type":"ContainerStarted","Data":"661416509ec346b08acdf43913ab62aa2887fe0fd3debcd709eeba1b09d6406d"} Jul 08 02:58:48.897749 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.897730 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9zm24" event={"ID":"b08cdccc-fc17-4ddf-9140-078331f5e052","Type":"ContainerStarted","Data":"4aa98d8d6e61bdd89c2228a66132cbc9018a6b2661cd12d143ad14f741c96557"} Jul 08 02:58:48.898750 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.898731 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" event={"ID":"4040905f-871b-470e-a30e-c52ec426b0de","Type":"ContainerStarted","Data":"6e02703e7bfd3579d12f5a14b834c2cf45c22605fc5c793733d50a76623d3e40"} Jul 08 02:58:48.899853 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.899835 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal" event={"ID":"233d4ffcc88e32f0bc8c7cf42f125937","Type":"ContainerStarted","Data":"4937ea4dfa6ce3e109a93ed1a82ab2024e9ef11a9ecebb5227f097617a352e3b"} Jul 08 02:58:48.905099 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:48.905056 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-70.ec2.internal" podStartSLOduration=2.905045787 podStartE2EDuration="2.905045787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:58:48.90500888 +0000 UTC m=+3.562384753" watchObservedRunningTime="2026-07-08 02:58:48.905045787 +0000 UTC m=+3.562421660" Jul 08 02:58:49.331152 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:49.331111 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f5283b1-208f-415c-819f-c3e479102863-metrics-certs\") pod \"network-metrics-daemon-lsmbz\" (UID: \"8f5283b1-208f-415c-819f-c3e479102863\") " pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:58:49.331354 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:49.331279 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:58:49.331354 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:49.331344 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f5283b1-208f-415c-819f-c3e479102863-metrics-certs podName:8f5283b1-208f-415c-819f-c3e479102863 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:51.331326192 +0000 UTC m=+5.988702046 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f5283b1-208f-415c-819f-c3e479102863-metrics-certs") pod "network-metrics-daemon-lsmbz" (UID: "8f5283b1-208f-415c-819f-c3e479102863") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:58:49.432094 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:49.432026 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6e66819-f708-4e22-81d4-46adeef611e0-original-pull-secret\") pod \"global-pull-secret-syncer-54p5l\" (UID: \"a6e66819-f708-4e22-81d4-46adeef611e0\") " pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:58:49.432094 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:49.432078 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh64r\" (UniqueName: \"kubernetes.io/projected/36d7c073-6c29-456f-a540-34e61812ac39-kube-api-access-nh64r\") pod \"network-check-target-hldxf\" (UID: \"36d7c073-6c29-456f-a540-34e61812ac39\") " pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:58:49.432333 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:49.432243 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 02:58:49.432333 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:49.432261 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 02:58:49.432333 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:49.432273 2564 projected.go:196] Error preparing data for projected volume kube-api-access-nh64r for pod openshift-network-diagnostics/network-check-target-hldxf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:58:49.432333 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:49.432330 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36d7c073-6c29-456f-a540-34e61812ac39-kube-api-access-nh64r podName:36d7c073-6c29-456f-a540-34e61812ac39 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:51.432312903 +0000 UTC m=+6.089688755 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nh64r" (UniqueName: "kubernetes.io/projected/36d7c073-6c29-456f-a540-34e61812ac39-kube-api-access-nh64r") pod "network-check-target-hldxf" (UID: "36d7c073-6c29-456f-a540-34e61812ac39") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:58:49.432817 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:49.432775 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 02:58:49.432884 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:49.432830 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e66819-f708-4e22-81d4-46adeef611e0-original-pull-secret podName:a6e66819-f708-4e22-81d4-46adeef611e0 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:51.432816271 +0000 UTC m=+6.090192148 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6e66819-f708-4e22-81d4-46adeef611e0-original-pull-secret") pod "global-pull-secret-syncer-54p5l" (UID: "a6e66819-f708-4e22-81d4-46adeef611e0") : object "kube-system"/"original-pull-secret" not registered Jul 08 02:58:49.743093 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:49.743016 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 02:53:47 +0000 UTC" deadline="2027-12-17 02:51:13.581657635 +0000 UTC" Jul 08 02:58:49.743093 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:49.743082 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12647h52m23.838579992s" Jul 08 02:58:49.883284 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:49.883249 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:58:49.883476 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:49.883387 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hldxf" podUID="36d7c073-6c29-456f-a540-34e61812ac39" Jul 08 02:58:49.883818 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:49.883795 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:58:49.883940 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:49.883901 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54p5l" podUID="a6e66819-f708-4e22-81d4-46adeef611e0" Jul 08 02:58:49.913951 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:49.913222 2564 generic.go:356] "Generic (PLEG): container finished" podID="233d4ffcc88e32f0bc8c7cf42f125937" containerID="4937ea4dfa6ce3e109a93ed1a82ab2024e9ef11a9ecebb5227f097617a352e3b" exitCode=0 Jul 08 02:58:49.914555 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:49.914155 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal" event={"ID":"233d4ffcc88e32f0bc8c7cf42f125937","Type":"ContainerDied","Data":"4937ea4dfa6ce3e109a93ed1a82ab2024e9ef11a9ecebb5227f097617a352e3b"} Jul 08 02:58:50.881356 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:50.880430 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:58:50.881356 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:50.880571 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lsmbz" podUID="8f5283b1-208f-415c-819f-c3e479102863" Jul 08 02:58:50.921844 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:50.921806 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal" event={"ID":"233d4ffcc88e32f0bc8c7cf42f125937","Type":"ContainerStarted","Data":"a6932cc5c57a2b62e87396b24de1be3fed6d9a7a11afcd2b3d22038942b27463"} Jul 08 02:58:50.938088 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:50.938029 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-70.ec2.internal" podStartSLOduration=4.937996576 podStartE2EDuration="4.937996576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:58:50.937865641 +0000 UTC m=+5.595241515" watchObservedRunningTime="2026-07-08 02:58:50.937996576 +0000 UTC m=+5.595372449" Jul 08 02:58:51.350199 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:51.350109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f5283b1-208f-415c-819f-c3e479102863-metrics-certs\") pod \"network-metrics-daemon-lsmbz\" (UID: \"8f5283b1-208f-415c-819f-c3e479102863\") " pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:58:51.350367 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:51.350271 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:58:51.350367 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:51.350332 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f5283b1-208f-415c-819f-c3e479102863-metrics-certs podName:8f5283b1-208f-415c-819f-c3e479102863 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:55.35031492 +0000 UTC m=+10.007690773 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f5283b1-208f-415c-819f-c3e479102863-metrics-certs") pod "network-metrics-daemon-lsmbz" (UID: "8f5283b1-208f-415c-819f-c3e479102863") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:58:51.451456 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:51.451415 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6e66819-f708-4e22-81d4-46adeef611e0-original-pull-secret\") pod \"global-pull-secret-syncer-54p5l\" (UID: \"a6e66819-f708-4e22-81d4-46adeef611e0\") " pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:58:51.451617 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:51.451472 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh64r\" (UniqueName: \"kubernetes.io/projected/36d7c073-6c29-456f-a540-34e61812ac39-kube-api-access-nh64r\") pod \"network-check-target-hldxf\" (UID: \"36d7c073-6c29-456f-a540-34e61812ac39\") " pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:58:51.451675 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:51.451651 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 02:58:51.451675 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:51.451670 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 02:58:51.451794 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:51.451682 2564 projected.go:196] Error preparing data for projected volume kube-api-access-nh64r for pod openshift-network-diagnostics/network-check-target-hldxf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:58:51.451794 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:51.451740 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36d7c073-6c29-456f-a540-34e61812ac39-kube-api-access-nh64r podName:36d7c073-6c29-456f-a540-34e61812ac39 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:55.451722386 +0000 UTC m=+10.109098251 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nh64r" (UniqueName: "kubernetes.io/projected/36d7c073-6c29-456f-a540-34e61812ac39-kube-api-access-nh64r") pod "network-check-target-hldxf" (UID: "36d7c073-6c29-456f-a540-34e61812ac39") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:58:51.452174 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:51.452151 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 02:58:51.452247 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:51.452217 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e66819-f708-4e22-81d4-46adeef611e0-original-pull-secret podName:a6e66819-f708-4e22-81d4-46adeef611e0 nodeName:}" failed. No retries permitted until 2026-07-08 02:58:55.452202967 +0000 UTC m=+10.109578820 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6e66819-f708-4e22-81d4-46adeef611e0-original-pull-secret") pod "global-pull-secret-syncer-54p5l" (UID: "a6e66819-f708-4e22-81d4-46adeef611e0") : object "kube-system"/"original-pull-secret" not registered Jul 08 02:58:51.880477 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:51.880441 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:58:51.880647 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:51.880582 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54p5l" podUID="a6e66819-f708-4e22-81d4-46adeef611e0" Jul 08 02:58:51.881124 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:51.881083 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:58:51.881220 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:51.881187 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hldxf" podUID="36d7c073-6c29-456f-a540-34e61812ac39" Jul 08 02:58:52.881093 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:52.881053 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:58:52.881549 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:52.881190 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lsmbz" podUID="8f5283b1-208f-415c-819f-c3e479102863" Jul 08 02:58:53.880807 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:53.880575 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:58:53.881004 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:53.880617 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:58:53.881004 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:53.880935 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54p5l" podUID="a6e66819-f708-4e22-81d4-46adeef611e0" Jul 08 02:58:53.881122 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:53.880999 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hldxf" podUID="36d7c073-6c29-456f-a540-34e61812ac39" Jul 08 02:58:54.880900 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:54.880848 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:58:54.881101 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:54.881061 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lsmbz" podUID="8f5283b1-208f-415c-819f-c3e479102863" Jul 08 02:58:55.386660 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:55.386617 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f5283b1-208f-415c-819f-c3e479102863-metrics-certs\") pod \"network-metrics-daemon-lsmbz\" (UID: \"8f5283b1-208f-415c-819f-c3e479102863\") " pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:58:55.387167 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:55.386748 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:58:55.387167 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:55.386819 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f5283b1-208f-415c-819f-c3e479102863-metrics-certs podName:8f5283b1-208f-415c-819f-c3e479102863 nodeName:}" failed. No retries permitted until 2026-07-08 02:59:03.386803823 +0000 UTC m=+18.044179672 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f5283b1-208f-415c-819f-c3e479102863-metrics-certs") pod "network-metrics-daemon-lsmbz" (UID: "8f5283b1-208f-415c-819f-c3e479102863") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:58:55.487564 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:55.486998 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6e66819-f708-4e22-81d4-46adeef611e0-original-pull-secret\") pod \"global-pull-secret-syncer-54p5l\" (UID: \"a6e66819-f708-4e22-81d4-46adeef611e0\") " pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:58:55.487564 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:55.487052 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh64r\" (UniqueName: \"kubernetes.io/projected/36d7c073-6c29-456f-a540-34e61812ac39-kube-api-access-nh64r\") pod \"network-check-target-hldxf\" (UID: \"36d7c073-6c29-456f-a540-34e61812ac39\") " pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:58:55.487564 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:55.487172 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 02:58:55.487564 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:55.487185 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 02:58:55.487564 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:55.487206 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 02:58:55.487564 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:55.487219 2564 projected.go:196] Error preparing data for projected volume kube-api-access-nh64r for pod openshift-network-diagnostics/network-check-target-hldxf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:58:55.487564 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:55.487255 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e66819-f708-4e22-81d4-46adeef611e0-original-pull-secret podName:a6e66819-f708-4e22-81d4-46adeef611e0 nodeName:}" failed. No retries permitted until 2026-07-08 02:59:03.487234687 +0000 UTC m=+18.144610542 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6e66819-f708-4e22-81d4-46adeef611e0-original-pull-secret") pod "global-pull-secret-syncer-54p5l" (UID: "a6e66819-f708-4e22-81d4-46adeef611e0") : object "kube-system"/"original-pull-secret" not registered Jul 08 02:58:55.487564 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:55.487276 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36d7c073-6c29-456f-a540-34e61812ac39-kube-api-access-nh64r podName:36d7c073-6c29-456f-a540-34e61812ac39 nodeName:}" failed. No retries permitted until 2026-07-08 02:59:03.487265439 +0000 UTC m=+18.144641302 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nh64r" (UniqueName: "kubernetes.io/projected/36d7c073-6c29-456f-a540-34e61812ac39-kube-api-access-nh64r") pod "network-check-target-hldxf" (UID: "36d7c073-6c29-456f-a540-34e61812ac39") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:58:55.881542 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:55.881506 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:58:55.881717 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:55.881617 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hldxf" podUID="36d7c073-6c29-456f-a540-34e61812ac39" Jul 08 02:58:55.889185 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:55.889159 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:58:55.889550 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:55.889525 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54p5l" podUID="a6e66819-f708-4e22-81d4-46adeef611e0" Jul 08 02:58:56.881354 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:56.880815 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:58:56.881354 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:56.880981 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lsmbz" podUID="8f5283b1-208f-415c-819f-c3e479102863" Jul 08 02:58:57.880948 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:57.880890 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:58:57.881134 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:57.880907 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:58:57.881134 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:57.881039 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54p5l" podUID="a6e66819-f708-4e22-81d4-46adeef611e0" Jul 08 02:58:57.881134 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:57.881119 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hldxf" podUID="36d7c073-6c29-456f-a540-34e61812ac39" Jul 08 02:58:58.880346 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:58.880309 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:58:58.880816 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:58.880458 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lsmbz" podUID="8f5283b1-208f-415c-819f-c3e479102863" Jul 08 02:58:59.880609 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:59.880570 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:58:59.881029 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:58:59.880585 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:58:59.881029 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:59.880695 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54p5l" podUID="a6e66819-f708-4e22-81d4-46adeef611e0" Jul 08 02:58:59.881029 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:58:59.880736 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hldxf" podUID="36d7c073-6c29-456f-a540-34e61812ac39" Jul 08 02:59:00.881055 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:00.881022 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:59:00.881458 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:00.881149 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lsmbz" podUID="8f5283b1-208f-415c-819f-c3e479102863" Jul 08 02:59:01.880503 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:01.880469 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:59:01.880665 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:01.880471 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:59:01.880665 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:01.880603 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54p5l" podUID="a6e66819-f708-4e22-81d4-46adeef611e0" Jul 08 02:59:01.880799 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:01.880658 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hldxf" podUID="36d7c073-6c29-456f-a540-34e61812ac39" Jul 08 02:59:02.880825 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:02.880788 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:59:02.881290 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:02.880959 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lsmbz" podUID="8f5283b1-208f-415c-819f-c3e479102863" Jul 08 02:59:03.451620 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:03.451575 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f5283b1-208f-415c-819f-c3e479102863-metrics-certs\") pod \"network-metrics-daemon-lsmbz\" (UID: \"8f5283b1-208f-415c-819f-c3e479102863\") " pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:59:03.451791 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:03.451706 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:59:03.451791 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:03.451768 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f5283b1-208f-415c-819f-c3e479102863-metrics-certs podName:8f5283b1-208f-415c-819f-c3e479102863 nodeName:}" failed. No retries permitted until 2026-07-08 02:59:19.45174965 +0000 UTC m=+34.109125506 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f5283b1-208f-415c-819f-c3e479102863-metrics-certs") pod "network-metrics-daemon-lsmbz" (UID: "8f5283b1-208f-415c-819f-c3e479102863") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:59:03.552048 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:03.552009 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6e66819-f708-4e22-81d4-46adeef611e0-original-pull-secret\") pod \"global-pull-secret-syncer-54p5l\" (UID: \"a6e66819-f708-4e22-81d4-46adeef611e0\") " pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:59:03.552048 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:03.552049 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh64r\" (UniqueName: \"kubernetes.io/projected/36d7c073-6c29-456f-a540-34e61812ac39-kube-api-access-nh64r\") pod \"network-check-target-hldxf\" (UID: \"36d7c073-6c29-456f-a540-34e61812ac39\") " pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:59:03.552236 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:03.552169 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 02:59:03.552236 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:03.552176 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 02:59:03.552236 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:03.552198 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 02:59:03.552236 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:03.552207 2564 projected.go:196] Error preparing data for projected volume kube-api-access-nh64r for pod openshift-network-diagnostics/network-check-target-hldxf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:59:03.552236 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:03.552237 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e66819-f708-4e22-81d4-46adeef611e0-original-pull-secret podName:a6e66819-f708-4e22-81d4-46adeef611e0 nodeName:}" failed. No retries permitted until 2026-07-08 02:59:19.552221705 +0000 UTC m=+34.209597554 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6e66819-f708-4e22-81d4-46adeef611e0-original-pull-secret") pod "global-pull-secret-syncer-54p5l" (UID: "a6e66819-f708-4e22-81d4-46adeef611e0") : object "kube-system"/"original-pull-secret" not registered Jul 08 02:59:03.552392 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:03.552253 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36d7c073-6c29-456f-a540-34e61812ac39-kube-api-access-nh64r podName:36d7c073-6c29-456f-a540-34e61812ac39 nodeName:}" failed. No retries permitted until 2026-07-08 02:59:19.552246574 +0000 UTC m=+34.209622426 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nh64r" (UniqueName: "kubernetes.io/projected/36d7c073-6c29-456f-a540-34e61812ac39-kube-api-access-nh64r") pod "network-check-target-hldxf" (UID: "36d7c073-6c29-456f-a540-34e61812ac39") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:59:03.880973 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:03.880940 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:59:03.881414 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:03.880939 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:59:03.881414 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:03.881066 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54p5l" podUID="a6e66819-f708-4e22-81d4-46adeef611e0" Jul 08 02:59:03.881414 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:03.881151 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hldxf" podUID="36d7c073-6c29-456f-a540-34e61812ac39" Jul 08 02:59:04.880571 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:04.880532 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:59:04.880733 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:04.880681 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lsmbz" podUID="8f5283b1-208f-415c-819f-c3e479102863" Jul 08 02:59:05.881854 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:05.881807 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:59:05.882298 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:05.881899 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:59:05.882298 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:05.881948 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54p5l" podUID="a6e66819-f708-4e22-81d4-46adeef611e0" Jul 08 02:59:05.882298 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:05.882030 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hldxf" podUID="36d7c073-6c29-456f-a540-34e61812ac39" Jul 08 02:59:06.880845 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:06.880796 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:59:06.881041 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:06.880967 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lsmbz" podUID="8f5283b1-208f-415c-819f-c3e479102863" Jul 08 02:59:07.880735 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:07.880564 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:59:07.881662 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:07.880656 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:59:07.881662 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:07.880810 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54p5l" podUID="a6e66819-f708-4e22-81d4-46adeef611e0" Jul 08 02:59:07.881662 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:07.880887 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hldxf" podUID="36d7c073-6c29-456f-a540-34e61812ac39" Jul 08 02:59:07.958268 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:07.958242 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4ccc2" event={"ID":"6c992baa-d9c2-4241-9c52-9c930569f591","Type":"ContainerStarted","Data":"58693284ac91f374cf76a0e4400a8fd01a6d42bd9c4647ab8e9501594837fa12"} Jul 08 02:59:07.959819 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:07.959790 2564 generic.go:356] "Generic (PLEG): container finished" podID="fce35c6b-7b91-4a96-8f3f-4227c1ef6127" containerID="174c615e3a8e7426f0517d9a491661d2be21d01ee1e9a2add3dc9e1e0e6b5258" exitCode=0 Jul 08 02:59:07.960119 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:07.959822 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwl2s" event={"ID":"fce35c6b-7b91-4a96-8f3f-4227c1ef6127","Type":"ContainerDied","Data":"174c615e3a8e7426f0517d9a491661d2be21d01ee1e9a2add3dc9e1e0e6b5258"} Jul 08 02:59:07.962006 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:07.961667 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8jg82" event={"ID":"2d241eb6-7dbd-4ad7-86ef-6ba3fd7bf61c","Type":"ContainerStarted","Data":"45e8ed53f203190ab1b16d427f275c922f4bc3e508e80a55a0bb506626781a82"} Jul 08 02:59:07.963309 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:07.963278 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6" event={"ID":"3d770ab2-b9c7-463f-bdf8-1e6249e18ea4","Type":"ContainerStarted","Data":"e3cdee3c3aab43b0eb4dcf2d0517f0978de14ffbb35722d61a38c49ea2f698ac"} Jul 08 02:59:07.965816 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:07.965485 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7v4gn" event={"ID":"97c25edf-e111-4f3c-933f-9e45c6926b2c","Type":"ContainerStarted","Data":"994804fce2af8ccd20b24655af9d01aabe59c758e191e37ee9d8827aeab2957a"} Jul 08 02:59:07.968049 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:07.968026 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b25jn" event={"ID":"570103f4-bd12-4dd7-813a-7d3a2028ff68","Type":"ContainerStarted","Data":"4dbbbbe0303444b68b551d712d403f7591cf5e064620df17f8cf6a21c4a4ab61"} Jul 08 02:59:07.969480 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:07.969460 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9zm24" event={"ID":"b08cdccc-fc17-4ddf-9140-078331f5e052","Type":"ContainerStarted","Data":"709d6d0471c077680406a4fc7ec09bdcc5d90e98df046b9a3c636617d0368526"} Jul 08 02:59:07.971370 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:07.971351 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" event={"ID":"4040905f-871b-470e-a30e-c52ec426b0de","Type":"ContainerStarted","Data":"e5f1cdd69003aa83a84ea49dcd7e138a14aac262e9a0ea4ca2879aa51e7df3c6"} Jul 08 02:59:07.971465 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:07.971421 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" event={"ID":"4040905f-871b-470e-a30e-c52ec426b0de","Type":"ContainerStarted","Data":"f6a7b26845e5fcd4c9e2ba07b436f126493b6d22a8e3d5b86d4d3e46b5d08d6b"} Jul 08 02:59:07.971527 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:07.971473 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" event={"ID":"4040905f-871b-470e-a30e-c52ec426b0de","Type":"ContainerStarted","Data":"29f71218cb47340c7e8031c076ed8e75173b7c1fe87806bdd18e0a6904f43699"} Jul 08 02:59:07.977755 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:07.977717 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4ccc2" podStartSLOduration=12.262415963 podStartE2EDuration="22.977705423s" totalImagesPullingTime="10.71528946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:07.977063273 +0000 UTC m=+22.634439146" watchObservedRunningTime="2026-07-08 02:59:07.977705423 +0000 UTC m=+22.635081294" Jul 08 02:59:07.992329 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:07.992291 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7v4gn" podStartSLOduration=4.075396115 podStartE2EDuration="22.992275638s" totalImagesPullingTime="18.916879523s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:07.992068569 +0000 UTC m=+22.649444440" watchObservedRunningTime="2026-07-08 02:59:07.992275638 +0000 UTC m=+22.649651509" Jul 08 02:59:07.998030 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:07.998007 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:08.053295 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:08.053246 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-b25jn" podStartSLOduration=4.080219137 podStartE2EDuration="23.053228212s" totalImagesPullingTime="18.973009075s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:08.053198971 +0000 UTC m=+22.710574843" watchObservedRunningTime="2026-07-08 02:59:08.053228212 +0000 UTC m=+22.710604077" Jul 08 02:59:08.053749 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:08.053723 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8jg82" podStartSLOduration=4.139217402 podStartE2EDuration="23.053715827s" totalImagesPullingTime="18.914498425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:08.033983416 +0000 UTC m=+22.691359288" watchObservedRunningTime="2026-07-08 02:59:08.053715827 +0000 UTC m=+22.711091698" Jul 08 02:59:08.071370 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:08.071276 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9zm24" podStartSLOduration=4.147586915 podStartE2EDuration="23.071260247s" totalImagesPullingTime="18.923673332s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:08.070934655 +0000 UTC m=+22.728310520" watchObservedRunningTime="2026-07-08 02:59:08.071260247 +0000 UTC m=+22.728636118" Jul 08 02:59:08.353078 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:08.353002 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:08.748392 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:08.748350 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:08.880837 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:08.880807 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:59:08.881310 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:08.880954 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lsmbz" podUID="8f5283b1-208f-415c-819f-c3e479102863" Jul 08 02:59:08.943011 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:08.942973 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:08.976974 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:08.976940 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b6kz7" event={"ID":"ab708362-5418-4355-936b-870a73469b42","Type":"ContainerStarted","Data":"54816f4eb51cbdbe3269fcbe1d3939e2dd53adc3ad930ae5f07622e18d8b742e"} Jul 08 02:59:08.979778 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:08.979745 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" event={"ID":"4040905f-871b-470e-a30e-c52ec426b0de","Type":"ContainerStarted","Data":"039ccb3a6318d64609a4c8b28f1b5af7c1f54b1d111e50207e6efa17e4dab527"} Jul 08 02:59:08.979931 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:08.979831 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" event={"ID":"4040905f-871b-470e-a30e-c52ec426b0de","Type":"ContainerStarted","Data":"68d642e8b45ee5ce2dcc147eeed97c8359b79ad54c3cf1e3d281fc4505f348ad"} Jul 08 02:59:08.979931 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:08.979871 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" event={"ID":"4040905f-871b-470e-a30e-c52ec426b0de","Type":"ContainerStarted","Data":"046c516403db2454e1ad650c6c1951c419e20877a58366a89ace578b18e888fe"} Jul 08 02:59:08.992059 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:08.992011 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-b6kz7" podStartSLOduration=5.075343806 podStartE2EDuration="23.991998501s" totalImagesPullingTime="18.916654695s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:08.991435102 +0000 UTC m=+23.648810974" watchObservedRunningTime="2026-07-08 02:59:08.991998501 +0000 UTC m=+23.649374371" Jul 08 02:59:08.996647 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:08.996624 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4ccc2" Jul 08 02:59:08.997223 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:08.997207 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4ccc2" Jul 08 02:59:09.322352 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:09.322325 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 08 02:59:09.342422 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:09.342390 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:09.780661 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:09.780560 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T02:59:09.322346144Z","UUID":"969a662f-6209-4165-b645-d9819a8ee56c","Handler":null,"Name":"","Endpoint":""} Jul 08 02:59:09.782334 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:09.782304 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 08 02:59:09.782454 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:09.782347 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 08 02:59:09.880894 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:09.880861 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:59:09.881656 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:09.880861 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:59:09.881656 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:09.881009 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hldxf" podUID="36d7c073-6c29-456f-a540-34e61812ac39" Jul 08 02:59:09.881656 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:09.881430 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54p5l" podUID="a6e66819-f708-4e22-81d4-46adeef611e0" Jul 08 02:59:09.983414 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:09.983374 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6" event={"ID":"3d770ab2-b9c7-463f-bdf8-1e6249e18ea4","Type":"ContainerStarted","Data":"287bcba68c649af110131ce5abd38293bd7ede2984d8e93ab8ea124fb60a5610"} Jul 08 02:59:09.983665 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:09.983648 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4ccc2" Jul 08 02:59:09.984150 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:09.984131 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4ccc2" Jul 08 02:59:10.147311 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:10.147267 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:10.538139 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:10.538103 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:10.640082 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:10.640048 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:10.743587 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:10.743474 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:10.881105 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:10.881077 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:59:10.881503 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:10.881206 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lsmbz" podUID="8f5283b1-208f-415c-819f-c3e479102863" Jul 08 02:59:10.942432 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:10.942399 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:10.990126 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:10.989710 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6" event={"ID":"3d770ab2-b9c7-463f-bdf8-1e6249e18ea4","Type":"ContainerStarted","Data":"03317d5ef023d2d68ad3918d8d3cd4895d6b25e7b491f2e724fad7b1a793302d"} Jul 08 02:59:10.993810 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:10.993747 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" event={"ID":"4040905f-871b-470e-a30e-c52ec426b0de","Type":"ContainerStarted","Data":"bdd860b1063d8576f9dceb75e84d81ba0b8dec6e8a63927758f940afb1c43f9e"} Jul 08 02:59:11.153621 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:11.153590 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:11.880515 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:11.880479 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:59:11.880687 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:11.880479 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:59:11.880687 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:11.880614 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hldxf" podUID="36d7c073-6c29-456f-a540-34e61812ac39" Jul 08 02:59:11.880687 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:11.880671 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54p5l" podUID="a6e66819-f708-4e22-81d4-46adeef611e0" Jul 08 02:59:11.946305 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:11.946269 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:12.017375 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:12.017327 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l5sw6" podStartSLOduration=5.182619498 podStartE2EDuration="27.01731122s" totalImagesPullingTime="21.834691722s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:58:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:12.016709851 +0000 UTC m=+26.674085723" watchObservedRunningTime="2026-07-08 02:59:12.01731122 +0000 UTC m=+26.674687136" Jul 08 02:59:12.152565 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:12.152481 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:12.360570 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:12.360540 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:12.601127 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:12.601044 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:12.880096 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:12.880058 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:59:12.880297 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:12.880209 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lsmbz" podUID="8f5283b1-208f-415c-819f-c3e479102863" Jul 08 02:59:13.000682 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:13.000654 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:13.127523 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:13.127488 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:13.226550 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:13.226454 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:13.283731 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:13.283701 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:13.460370 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:13.460193 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:13.866059 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:13.865978 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:13.880482 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:13.880448 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:59:13.880482 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:13.880469 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:59:13.880665 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:13.880592 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54p5l" podUID="a6e66819-f708-4e22-81d4-46adeef611e0" Jul 08 02:59:13.880747 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:13.880692 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hldxf" podUID="36d7c073-6c29-456f-a540-34e61812ac39" Jul 08 02:59:13.967433 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:13.967399 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:14.004989 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:14.004950 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" event={"ID":"4040905f-871b-470e-a30e-c52ec426b0de","Type":"ContainerStarted","Data":"05bfcd91c1a094644a82a0df477ab33242a12187a442810246f5003673882141"} Jul 08 02:59:14.005445 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:14.005317 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:59:14.005445 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:14.005416 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:59:14.007020 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:14.006987 2564 generic.go:356] "Generic (PLEG): container finished" podID="fce35c6b-7b91-4a96-8f3f-4227c1ef6127" containerID="f48041e1434ec38221b0331a6d8c278efe8a991e000e4c476e22853e158d372c" exitCode=0 Jul 08 02:59:14.007132 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:14.007024 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwl2s" event={"ID":"fce35c6b-7b91-4a96-8f3f-4227c1ef6127","Type":"ContainerDied","Data":"f48041e1434ec38221b0331a6d8c278efe8a991e000e4c476e22853e158d372c"} Jul 08 02:59:14.024376 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:14.024354 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:59:14.069779 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:14.069731 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" podStartSLOduration=10.079899299 podStartE2EDuration="29.069715892s" totalImagesPullingTime="18.989816593s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:14.037172868 +0000 UTC m=+28.694548737" watchObservedRunningTime="2026-07-08 02:59:14.069715892 +0000 UTC m=+28.727091763" Jul 08 02:59:14.079508 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:14.079472 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:14.166374 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:14.166339 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:14.495115 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:14.495036 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:14.670183 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:14.670149 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:14.880443 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:14.880419 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:59:14.880611 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:14.880535 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lsmbz" podUID="8f5283b1-208f-415c-819f-c3e479102863" Jul 08 02:59:15.009952 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:15.009907 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:59:15.025272 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:15.025246 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:59:15.105408 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:15.105370 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:15.496410 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:15.496169 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:15.881350 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:15.881319 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:59:15.881543 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:15.881410 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54p5l" podUID="a6e66819-f708-4e22-81d4-46adeef611e0" Jul 08 02:59:15.881543 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:15.881515 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:59:15.881676 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:15.881653 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hldxf" podUID="36d7c073-6c29-456f-a540-34e61812ac39" Jul 08 02:59:16.013554 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:16.013523 2564 generic.go:356] "Generic (PLEG): container finished" podID="fce35c6b-7b91-4a96-8f3f-4227c1ef6127" containerID="ffa340fc4dc741045812fb794f9c64cb384d302d3bf9bd5ed8b5075b8696ec6b" exitCode=0 Jul 08 02:59:16.013974 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:16.013617 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwl2s" event={"ID":"fce35c6b-7b91-4a96-8f3f-4227c1ef6127","Type":"ContainerDied","Data":"ffa340fc4dc741045812fb794f9c64cb384d302d3bf9bd5ed8b5075b8696ec6b"} Jul 08 02:59:16.880755 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:16.880719 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:59:16.881000 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:16.880833 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lsmbz" podUID="8f5283b1-208f-415c-819f-c3e479102863" Jul 08 02:59:17.880220 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:17.880181 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:59:17.880795 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:17.880188 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:59:17.880795 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:17.880300 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54p5l" podUID="a6e66819-f708-4e22-81d4-46adeef611e0" Jul 08 02:59:17.880795 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:17.880361 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hldxf" podUID="36d7c073-6c29-456f-a540-34e61812ac39" Jul 08 02:59:18.021180 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:18.021142 2564 generic.go:356] "Generic (PLEG): container finished" podID="fce35c6b-7b91-4a96-8f3f-4227c1ef6127" containerID="ba44b1e57f6eb33ad7738ccfe922037f2722111fa0b9648a0b551e23500bb21e" exitCode=0 Jul 08 02:59:18.021340 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:18.021189 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwl2s" event={"ID":"fce35c6b-7b91-4a96-8f3f-4227c1ef6127","Type":"ContainerDied","Data":"ba44b1e57f6eb33ad7738ccfe922037f2722111fa0b9648a0b551e23500bb21e"} Jul 08 02:59:18.881003 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:18.880967 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:59:18.881418 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:18.881115 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lsmbz" podUID="8f5283b1-208f-415c-819f-c3e479102863" Jul 08 02:59:19.473590 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:19.473546 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f5283b1-208f-415c-819f-c3e479102863-metrics-certs\") pod \"network-metrics-daemon-lsmbz\" (UID: \"8f5283b1-208f-415c-819f-c3e479102863\") " pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:59:19.473764 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:19.473676 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:59:19.473764 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:19.473742 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f5283b1-208f-415c-819f-c3e479102863-metrics-certs podName:8f5283b1-208f-415c-819f-c3e479102863 nodeName:}" failed. No retries permitted until 2026-07-08 02:59:51.473725775 +0000 UTC m=+66.131101637 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8f5283b1-208f-415c-819f-c3e479102863-metrics-certs") pod "network-metrics-daemon-lsmbz" (UID: "8f5283b1-208f-415c-819f-c3e479102863") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 02:59:19.574733 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:19.574694 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6e66819-f708-4e22-81d4-46adeef611e0-original-pull-secret\") pod \"global-pull-secret-syncer-54p5l\" (UID: \"a6e66819-f708-4e22-81d4-46adeef611e0\") " pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:59:19.574950 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:19.574741 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh64r\" (UniqueName: \"kubernetes.io/projected/36d7c073-6c29-456f-a540-34e61812ac39-kube-api-access-nh64r\") pod \"network-check-target-hldxf\" (UID: \"36d7c073-6c29-456f-a540-34e61812ac39\") " pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:59:19.574950 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:19.574877 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 02:59:19.574950 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:19.574935 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 02:59:19.575129 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:19.574957 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 02:59:19.575129 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:19.574964 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e66819-f708-4e22-81d4-46adeef611e0-original-pull-secret podName:a6e66819-f708-4e22-81d4-46adeef611e0 nodeName:}" failed. No retries permitted until 2026-07-08 02:59:51.574943446 +0000 UTC m=+66.232319310 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6e66819-f708-4e22-81d4-46adeef611e0-original-pull-secret") pod "global-pull-secret-syncer-54p5l" (UID: "a6e66819-f708-4e22-81d4-46adeef611e0") : object "kube-system"/"original-pull-secret" not registered Jul 08 02:59:19.575129 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:19.574999 2564 projected.go:196] Error preparing data for projected volume kube-api-access-nh64r for pod openshift-network-diagnostics/network-check-target-hldxf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:59:19.575129 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:19.575106 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36d7c073-6c29-456f-a540-34e61812ac39-kube-api-access-nh64r podName:36d7c073-6c29-456f-a540-34e61812ac39 nodeName:}" failed. No retries permitted until 2026-07-08 02:59:51.5750773 +0000 UTC m=+66.232453164 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nh64r" (UniqueName: "kubernetes.io/projected/36d7c073-6c29-456f-a540-34e61812ac39-kube-api-access-nh64r") pod "network-check-target-hldxf" (UID: "36d7c073-6c29-456f-a540-34e61812ac39") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 02:59:19.881115 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:19.881078 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:59:19.881538 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:19.881077 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:59:19.881538 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:19.881197 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hldxf" podUID="36d7c073-6c29-456f-a540-34e61812ac39" Jul 08 02:59:19.881538 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:19.881268 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54p5l" podUID="a6e66819-f708-4e22-81d4-46adeef611e0" Jul 08 02:59:19.958476 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:19.958427 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:20.016904 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:20.016836 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:20.881021 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:20.880991 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:59:20.881187 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:20.881128 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lsmbz" podUID="8f5283b1-208f-415c-819f-c3e479102863" Jul 08 02:59:21.881048 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:21.881008 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:59:21.881243 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:21.881132 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54p5l" podUID="a6e66819-f708-4e22-81d4-46adeef611e0" Jul 08 02:59:21.881243 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:21.881193 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:59:21.881749 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:21.881273 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hldxf" podUID="36d7c073-6c29-456f-a540-34e61812ac39" Jul 08 02:59:22.880844 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:22.880807 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:59:22.881056 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:22.880938 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lsmbz" podUID="8f5283b1-208f-415c-819f-c3e479102863" Jul 08 02:59:23.035778 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:23.035737 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwl2s" event={"ID":"fce35c6b-7b91-4a96-8f3f-4227c1ef6127","Type":"ContainerStarted","Data":"53c0f097dd9bd2c7b50f5843864f29f7dd4ff2b120496aa6b9bbae4d0fc37ae1"} Jul 08 02:59:23.880530 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:23.880493 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:59:23.880728 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:23.880498 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:59:23.880728 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:23.880596 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hldxf" podUID="36d7c073-6c29-456f-a540-34e61812ac39" Jul 08 02:59:23.880728 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:23.880668 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54p5l" podUID="a6e66819-f708-4e22-81d4-46adeef611e0" Jul 08 02:59:24.040189 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:24.040160 2564 generic.go:356] "Generic (PLEG): container finished" podID="fce35c6b-7b91-4a96-8f3f-4227c1ef6127" containerID="53c0f097dd9bd2c7b50f5843864f29f7dd4ff2b120496aa6b9bbae4d0fc37ae1" exitCode=0 Jul 08 02:59:24.040644 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:24.040211 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwl2s" event={"ID":"fce35c6b-7b91-4a96-8f3f-4227c1ef6127","Type":"ContainerDied","Data":"53c0f097dd9bd2c7b50f5843864f29f7dd4ff2b120496aa6b9bbae4d0fc37ae1"} Jul 08 02:59:24.099689 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:24.099660 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:24.197644 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:24.197591 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:24.297831 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:24.297794 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:24.398328 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:24.398228 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:24.880387 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:24.880359 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:59:24.880540 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:24.880480 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lsmbz" podUID="8f5283b1-208f-415c-819f-c3e479102863" Jul 08 02:59:25.045275 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:25.045241 2564 generic.go:356] "Generic (PLEG): container finished" podID="fce35c6b-7b91-4a96-8f3f-4227c1ef6127" containerID="4410b9685d2167faef97c79a008827aee58d7311b72d882053a37330185a0e93" exitCode=0 Jul 08 02:59:25.045663 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:25.045280 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwl2s" event={"ID":"fce35c6b-7b91-4a96-8f3f-4227c1ef6127","Type":"ContainerDied","Data":"4410b9685d2167faef97c79a008827aee58d7311b72d882053a37330185a0e93"} Jul 08 02:59:25.178702 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:25.178667 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:25.283096 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:25.283059 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:25.371317 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:25.371287 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:25.452498 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:25.452412 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:25.553402 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:25.553372 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:25.881732 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:25.881706 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:59:25.881908 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:25.881784 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:59:25.881908 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:25.881793 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hldxf" podUID="36d7c073-6c29-456f-a540-34e61812ac39" Jul 08 02:59:25.881908 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:25.881894 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54p5l" podUID="a6e66819-f708-4e22-81d4-46adeef611e0" Jul 08 02:59:26.051152 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:26.051116 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwl2s" event={"ID":"fce35c6b-7b91-4a96-8f3f-4227c1ef6127","Type":"ContainerStarted","Data":"895d014cbcf5517c1187cebc865522f98694b192fbb8f933aa6e086971d63285"} Jul 08 02:59:26.076422 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:26.076373 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lwl2s" podStartSLOduration=9.362088546 podStartE2EDuration="41.076358764s" totalImagesPullingTime="31.714270218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:58:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:26.074576034 +0000 UTC m=+40.731951902" watchObservedRunningTime="2026-07-08 02:59:26.076358764 +0000 UTC m=+40.733734635" Jul 08 02:59:26.459741 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:26.459473 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:26.559274 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:26.559223 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:26.577194 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:26.577162 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hldxf"] Jul 08 02:59:26.577350 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:26.577264 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:59:26.577393 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:26.577344 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hldxf" podUID="36d7c073-6c29-456f-a540-34e61812ac39" Jul 08 02:59:26.580004 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:26.579978 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-54p5l"] Jul 08 02:59:26.580215 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:26.580199 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:59:26.580377 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:26.580349 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54p5l" podUID="a6e66819-f708-4e22-81d4-46adeef611e0" Jul 08 02:59:26.580711 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:26.580689 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lsmbz"] Jul 08 02:59:26.580812 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:26.580798 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:59:26.580945 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:26.580905 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lsmbz" podUID="8f5283b1-208f-415c-819f-c3e479102863" Jul 08 02:59:26.865631 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:26.865602 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:26.953611 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:26.953586 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:27.116751 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:27.116664 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:27.880506 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:27.880466 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:59:27.880646 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:27.880513 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:59:27.880646 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:27.880598 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54p5l" podUID="a6e66819-f708-4e22-81d4-46adeef611e0" Jul 08 02:59:27.880738 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:27.880718 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lsmbz" podUID="8f5283b1-208f-415c-819f-c3e479102863" Jul 08 02:59:28.880427 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:28.880392 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:59:28.880799 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:28.880496 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hldxf" podUID="36d7c073-6c29-456f-a540-34e61812ac39" Jul 08 02:59:29.880527 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:29.880494 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:59:29.880885 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:29.880493 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:59:29.880885 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:29.880639 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lsmbz" podUID="8f5283b1-208f-415c-819f-c3e479102863" Jul 08 02:59:29.880885 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:29.880684 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-54p5l" podUID="a6e66819-f708-4e22-81d4-46adeef611e0" Jul 08 02:59:30.132086 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:30.131998 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:30.233655 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:30.233618 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:30.497694 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:30.497588 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:30.548417 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:30.548380 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:30.880493 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:30.880462 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:59:30.880662 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:30.880575 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hldxf" podUID="36d7c073-6c29-456f-a540-34e61812ac39" Jul 08 02:59:31.018849 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.018815 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:31.115613 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.115578 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:31.245765 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.245644 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:31.309607 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.309579 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:31.643974 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.643946 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-70.ec2.internal" event="NodeReady" Jul 08 02:59:31.644122 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.644077 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 08 02:59:31.691128 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.691098 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b49sr"] Jul 08 02:59:31.710884 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.710855 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ndv6v"] Jul 08 02:59:31.711065 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.710987 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b49sr" Jul 08 02:59:31.715443 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.715221 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 08 02:59:31.717394 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.717369 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 08 02:59:31.717394 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.717386 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kmjwn\"" Jul 08 02:59:31.717887 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.717585 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 08 02:59:31.727213 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.727191 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b49sr"] Jul 08 02:59:31.727308 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.727225 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ndv6v"] Jul 08 02:59:31.727308 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.727256 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rkrlf"] Jul 08 02:59:31.727308 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.727293 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ndv6v" Jul 08 02:59:31.730143 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.730117 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 08 02:59:31.730229 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.730158 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 08 02:59:31.730286 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.730224 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5d4df\"" Jul 08 02:59:31.730339 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.730321 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 08 02:59:31.730382 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.730352 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 08 02:59:31.746998 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.746964 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rkrlf"] Jul 08 02:59:31.747139 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.747033 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rkrlf" Jul 08 02:59:31.749833 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.749806 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 08 02:59:31.750267 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.750249 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 08 02:59:31.750354 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.750252 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6djh2\"" Jul 08 02:59:31.851463 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.851426 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:31.859456 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.859430 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4bd7623c-f229-43b7-aefd-339ec594cf2c-crio-socket\") pod \"insights-runtime-extractor-ndv6v\" (UID: \"4bd7623c-f229-43b7-aefd-339ec594cf2c\") " pod="openshift-insights/insights-runtime-extractor-ndv6v" Jul 08 02:59:31.859550 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.859466 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvx66\" (UniqueName: \"kubernetes.io/projected/cd436540-823b-4c37-9a20-b8ebdf0937b7-kube-api-access-jvx66\") pod \"dns-default-rkrlf\" (UID: \"cd436540-823b-4c37-9a20-b8ebdf0937b7\") " pod="openshift-dns/dns-default-rkrlf" Jul 08 02:59:31.859550 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.859501 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9489f6b7-c200-4e47-93f7-2f8a8e4be132-cert\") pod \"ingress-canary-b49sr\" (UID: \"9489f6b7-c200-4e47-93f7-2f8a8e4be132\") " pod="openshift-ingress-canary/ingress-canary-b49sr" Jul 08 02:59:31.859550 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.859526 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4bd7623c-f229-43b7-aefd-339ec594cf2c-data-volume\") pod \"insights-runtime-extractor-ndv6v\" (UID: \"4bd7623c-f229-43b7-aefd-339ec594cf2c\") " pod="openshift-insights/insights-runtime-extractor-ndv6v" Jul 08 02:59:31.859701 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.859598 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4bd7623c-f229-43b7-aefd-339ec594cf2c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ndv6v\" (UID: \"4bd7623c-f229-43b7-aefd-339ec594cf2c\") " pod="openshift-insights/insights-runtime-extractor-ndv6v" Jul 08 02:59:31.859701 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.859666 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd436540-823b-4c37-9a20-b8ebdf0937b7-tmp-dir\") pod \"dns-default-rkrlf\" (UID: \"cd436540-823b-4c37-9a20-b8ebdf0937b7\") " pod="openshift-dns/dns-default-rkrlf" Jul 08 02:59:31.859789 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.859703 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmxrt\" (UniqueName: \"kubernetes.io/projected/9489f6b7-c200-4e47-93f7-2f8a8e4be132-kube-api-access-tmxrt\") pod \"ingress-canary-b49sr\" (UID: \"9489f6b7-c200-4e47-93f7-2f8a8e4be132\") " pod="openshift-ingress-canary/ingress-canary-b49sr" Jul 08 02:59:31.859789 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.859730 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm8qz\" (UniqueName: \"kubernetes.io/projected/4bd7623c-f229-43b7-aefd-339ec594cf2c-kube-api-access-pm8qz\") pod \"insights-runtime-extractor-ndv6v\" (UID: \"4bd7623c-f229-43b7-aefd-339ec594cf2c\") " pod="openshift-insights/insights-runtime-extractor-ndv6v" Jul 08 02:59:31.859789 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.859768 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cd436540-823b-4c37-9a20-b8ebdf0937b7-config-volume\") pod \"dns-default-rkrlf\" (UID: \"cd436540-823b-4c37-9a20-b8ebdf0937b7\") " pod="openshift-dns/dns-default-rkrlf" Jul 08 02:59:31.859906 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.859814 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4bd7623c-f229-43b7-aefd-339ec594cf2c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ndv6v\" (UID: \"4bd7623c-f229-43b7-aefd-339ec594cf2c\") " pod="openshift-insights/insights-runtime-extractor-ndv6v" Jul 08 02:59:31.859906 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.859858 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd436540-823b-4c37-9a20-b8ebdf0937b7-metrics-tls\") pod \"dns-default-rkrlf\" (UID: \"cd436540-823b-4c37-9a20-b8ebdf0937b7\") " pod="openshift-dns/dns-default-rkrlf" Jul 08 02:59:31.880596 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.880570 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:59:31.880709 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.880570 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:59:31.883453 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.883428 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 02:59:31.883453 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.883449 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 02:59:31.883639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.883507 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fsx5r\"" Jul 08 02:59:31.961047 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.960956 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9489f6b7-c200-4e47-93f7-2f8a8e4be132-cert\") pod \"ingress-canary-b49sr\" (UID: \"9489f6b7-c200-4e47-93f7-2f8a8e4be132\") " pod="openshift-ingress-canary/ingress-canary-b49sr" Jul 08 02:59:31.961047 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.960994 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4bd7623c-f229-43b7-aefd-339ec594cf2c-data-volume\") pod \"insights-runtime-extractor-ndv6v\" (UID: \"4bd7623c-f229-43b7-aefd-339ec594cf2c\") " pod="openshift-insights/insights-runtime-extractor-ndv6v" Jul 08 02:59:31.961047 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.961014 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4bd7623c-f229-43b7-aefd-339ec594cf2c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ndv6v\" (UID: \"4bd7623c-f229-43b7-aefd-339ec594cf2c\") " pod="openshift-insights/insights-runtime-extractor-ndv6v" Jul 08 02:59:31.961047 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.961037 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd436540-823b-4c37-9a20-b8ebdf0937b7-tmp-dir\") pod \"dns-default-rkrlf\" (UID: \"cd436540-823b-4c37-9a20-b8ebdf0937b7\") " pod="openshift-dns/dns-default-rkrlf" Jul 08 02:59:31.961405 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.961065 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmxrt\" (UniqueName: \"kubernetes.io/projected/9489f6b7-c200-4e47-93f7-2f8a8e4be132-kube-api-access-tmxrt\") pod \"ingress-canary-b49sr\" (UID: \"9489f6b7-c200-4e47-93f7-2f8a8e4be132\") " pod="openshift-ingress-canary/ingress-canary-b49sr" Jul 08 02:59:31.961405 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.961092 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm8qz\" (UniqueName: \"kubernetes.io/projected/4bd7623c-f229-43b7-aefd-339ec594cf2c-kube-api-access-pm8qz\") pod \"insights-runtime-extractor-ndv6v\" (UID: \"4bd7623c-f229-43b7-aefd-339ec594cf2c\") " pod="openshift-insights/insights-runtime-extractor-ndv6v" Jul 08 02:59:31.961405 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.961180 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cd436540-823b-4c37-9a20-b8ebdf0937b7-config-volume\") pod \"dns-default-rkrlf\" (UID: \"cd436540-823b-4c37-9a20-b8ebdf0937b7\") " pod="openshift-dns/dns-default-rkrlf" Jul 08 02:59:31.961405 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.961226 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4bd7623c-f229-43b7-aefd-339ec594cf2c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ndv6v\" (UID: \"4bd7623c-f229-43b7-aefd-339ec594cf2c\") " pod="openshift-insights/insights-runtime-extractor-ndv6v" Jul 08 02:59:31.961405 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.961278 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd436540-823b-4c37-9a20-b8ebdf0937b7-metrics-tls\") pod \"dns-default-rkrlf\" (UID: \"cd436540-823b-4c37-9a20-b8ebdf0937b7\") " pod="openshift-dns/dns-default-rkrlf" Jul 08 02:59:31.961405 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.961308 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4bd7623c-f229-43b7-aefd-339ec594cf2c-crio-socket\") pod \"insights-runtime-extractor-ndv6v\" (UID: \"4bd7623c-f229-43b7-aefd-339ec594cf2c\") " pod="openshift-insights/insights-runtime-extractor-ndv6v" Jul 08 02:59:31.961405 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.961337 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvx66\" (UniqueName: \"kubernetes.io/projected/cd436540-823b-4c37-9a20-b8ebdf0937b7-kube-api-access-jvx66\") pod \"dns-default-rkrlf\" (UID: \"cd436540-823b-4c37-9a20-b8ebdf0937b7\") " pod="openshift-dns/dns-default-rkrlf" Jul 08 02:59:31.961729 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.961408 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4bd7623c-f229-43b7-aefd-339ec594cf2c-data-volume\") pod \"insights-runtime-extractor-ndv6v\" (UID: \"4bd7623c-f229-43b7-aefd-339ec594cf2c\") " pod="openshift-insights/insights-runtime-extractor-ndv6v" Jul 08 02:59:31.961729 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.961424 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd436540-823b-4c37-9a20-b8ebdf0937b7-tmp-dir\") pod \"dns-default-rkrlf\" (UID: \"cd436540-823b-4c37-9a20-b8ebdf0937b7\") " pod="openshift-dns/dns-default-rkrlf" Jul 08 02:59:31.961729 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.961554 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4bd7623c-f229-43b7-aefd-339ec594cf2c-crio-socket\") pod \"insights-runtime-extractor-ndv6v\" (UID: \"4bd7623c-f229-43b7-aefd-339ec594cf2c\") " pod="openshift-insights/insights-runtime-extractor-ndv6v" Jul 08 02:59:31.961849 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.961743 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4bd7623c-f229-43b7-aefd-339ec594cf2c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ndv6v\" (UID: \"4bd7623c-f229-43b7-aefd-339ec594cf2c\") " pod="openshift-insights/insights-runtime-extractor-ndv6v" Jul 08 02:59:31.961849 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.961777 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cd436540-823b-4c37-9a20-b8ebdf0937b7-config-volume\") pod \"dns-default-rkrlf\" (UID: \"cd436540-823b-4c37-9a20-b8ebdf0937b7\") " pod="openshift-dns/dns-default-rkrlf" Jul 08 02:59:31.965490 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.965462 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4bd7623c-f229-43b7-aefd-339ec594cf2c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ndv6v\" (UID: \"4bd7623c-f229-43b7-aefd-339ec594cf2c\") " pod="openshift-insights/insights-runtime-extractor-ndv6v" Jul 08 02:59:31.965490 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.965480 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd436540-823b-4c37-9a20-b8ebdf0937b7-metrics-tls\") pod \"dns-default-rkrlf\" (UID: \"cd436540-823b-4c37-9a20-b8ebdf0937b7\") " pod="openshift-dns/dns-default-rkrlf" Jul 08 02:59:31.965639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.965568 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9489f6b7-c200-4e47-93f7-2f8a8e4be132-cert\") pod \"ingress-canary-b49sr\" (UID: \"9489f6b7-c200-4e47-93f7-2f8a8e4be132\") " pod="openshift-ingress-canary/ingress-canary-b49sr" Jul 08 02:59:31.970557 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.970525 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm8qz\" (UniqueName: \"kubernetes.io/projected/4bd7623c-f229-43b7-aefd-339ec594cf2c-kube-api-access-pm8qz\") pod \"insights-runtime-extractor-ndv6v\" (UID: \"4bd7623c-f229-43b7-aefd-339ec594cf2c\") " pod="openshift-insights/insights-runtime-extractor-ndv6v" Jul 08 02:59:31.970727 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.970573 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvx66\" (UniqueName: \"kubernetes.io/projected/cd436540-823b-4c37-9a20-b8ebdf0937b7-kube-api-access-jvx66\") pod \"dns-default-rkrlf\" (UID: \"cd436540-823b-4c37-9a20-b8ebdf0937b7\") " pod="openshift-dns/dns-default-rkrlf" Jul 08 02:59:31.970727 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:31.970577 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmxrt\" (UniqueName: \"kubernetes.io/projected/9489f6b7-c200-4e47-93f7-2f8a8e4be132-kube-api-access-tmxrt\") pod \"ingress-canary-b49sr\" (UID: \"9489f6b7-c200-4e47-93f7-2f8a8e4be132\") " pod="openshift-ingress-canary/ingress-canary-b49sr" Jul 08 02:59:32.023577 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:32.023542 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b49sr" Jul 08 02:59:32.036750 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:32.036721 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ndv6v" Jul 08 02:59:32.055947 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:32.055890 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rkrlf" Jul 08 02:59:32.236973 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:32.236678 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b49sr"] Jul 08 02:59:32.239974 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:32.239952 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ndv6v"] Jul 08 02:59:32.241589 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:59:32.241565 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9489f6b7_c200_4e47_93f7_2f8a8e4be132.slice/crio-154e6000489b4a3a24041ca38bbbdd032b05f319f9d1c909d7f3a5303ec7e191 WatchSource:0}: Error finding container 154e6000489b4a3a24041ca38bbbdd032b05f319f9d1c909d7f3a5303ec7e191: Status 404 returned error can't find the container with id 154e6000489b4a3a24041ca38bbbdd032b05f319f9d1c909d7f3a5303ec7e191 Jul 08 02:59:32.242814 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:59:32.242786 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd7623c_f229_43b7_aefd_339ec594cf2c.slice/crio-ec4159b5e07bb8051ffa1adde3ad43275f0fa9a423ead661b194ca288761ece8 WatchSource:0}: Error finding container ec4159b5e07bb8051ffa1adde3ad43275f0fa9a423ead661b194ca288761ece8: Status 404 returned error can't find the container with id ec4159b5e07bb8051ffa1adde3ad43275f0fa9a423ead661b194ca288761ece8 Jul 08 02:59:32.253237 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:32.253209 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rkrlf"] Jul 08 02:59:32.256669 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:59:32.256645 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd436540_823b_4c37_9a20_b8ebdf0937b7.slice/crio-1d6bb1ae4dcfde2f8c69016231a883c6d025be03c12c77390a7503a6b1a2b0fb WatchSource:0}: Error finding container 1d6bb1ae4dcfde2f8c69016231a883c6d025be03c12c77390a7503a6b1a2b0fb: Status 404 returned error can't find the container with id 1d6bb1ae4dcfde2f8c69016231a883c6d025be03c12c77390a7503a6b1a2b0fb Jul 08 02:59:32.351533 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:32.351499 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:32.412899 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:32.412866 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:32.503779 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:32.503699 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:32.565612 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:32.565580 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:32.740479 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:32.740442 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:32.841750 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:32.841659 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:32.880653 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:32.880621 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:59:32.884529 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:32.884291 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 02:59:32.885406 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:32.885203 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 02:59:32.885406 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:32.885263 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v7mdx\"" Jul 08 02:59:33.070902 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:33.070825 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b49sr" event={"ID":"9489f6b7-c200-4e47-93f7-2f8a8e4be132","Type":"ContainerStarted","Data":"154e6000489b4a3a24041ca38bbbdd032b05f319f9d1c909d7f3a5303ec7e191"} Jul 08 02:59:33.072452 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:33.072417 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rkrlf" event={"ID":"cd436540-823b-4c37-9a20-b8ebdf0937b7","Type":"ContainerStarted","Data":"1d6bb1ae4dcfde2f8c69016231a883c6d025be03c12c77390a7503a6b1a2b0fb"} Jul 08 02:59:33.074474 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:33.074446 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ndv6v" event={"ID":"4bd7623c-f229-43b7-aefd-339ec594cf2c","Type":"ContainerStarted","Data":"ec4159b5e07bb8051ffa1adde3ad43275f0fa9a423ead661b194ca288761ece8"} Jul 08 02:59:35.273131 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:35.273094 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:35.307570 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:35.307539 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-74t5g"] Jul 08 02:59:35.332508 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:35.332470 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-74t5g"] Jul 08 02:59:35.332680 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:35.332586 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-74t5g" Jul 08 02:59:35.335503 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:35.335481 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 08 02:59:35.335656 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:35.335582 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-d54j9\"" Jul 08 02:59:35.335656 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:35.335591 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 08 02:59:35.336749 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:35.336731 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 08 02:59:35.336749 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:35.336743 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 08 02:59:35.336886 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:35.336735 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 08 02:59:35.358348 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:35.358316 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:35.436007 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:35.435976 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:35.489239 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:35.489203 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mbqx\" (UniqueName: \"kubernetes.io/projected/e8f8dbad-7d14-4946-a94e-507e7405141b-kube-api-access-6mbqx\") pod \"prometheus-operator-567f7b8b7c-74t5g\" (UID: \"e8f8dbad-7d14-4946-a94e-507e7405141b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-74t5g" Jul 08 02:59:35.489428 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:35.489252 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8f8dbad-7d14-4946-a94e-507e7405141b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-74t5g\" (UID: \"e8f8dbad-7d14-4946-a94e-507e7405141b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-74t5g" Jul 08 02:59:35.489428 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:35.489342 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8f8dbad-7d14-4946-a94e-507e7405141b-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-74t5g\" (UID: \"e8f8dbad-7d14-4946-a94e-507e7405141b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-74t5g" Jul 08 02:59:35.489428 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:35.489364 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8f8dbad-7d14-4946-a94e-507e7405141b-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-74t5g\" (UID: \"e8f8dbad-7d14-4946-a94e-507e7405141b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-74t5g" Jul 08 02:59:35.589962 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:35.589866 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mbqx\" (UniqueName: \"kubernetes.io/projected/e8f8dbad-7d14-4946-a94e-507e7405141b-kube-api-access-6mbqx\") pod \"prometheus-operator-567f7b8b7c-74t5g\" (UID: \"e8f8dbad-7d14-4946-a94e-507e7405141b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-74t5g" Jul 08 02:59:35.589962 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:35.589940 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8f8dbad-7d14-4946-a94e-507e7405141b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-74t5g\" (UID: \"e8f8dbad-7d14-4946-a94e-507e7405141b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-74t5g" Jul 08 02:59:35.590184 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:35.590022 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8f8dbad-7d14-4946-a94e-507e7405141b-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-74t5g\" (UID: \"e8f8dbad-7d14-4946-a94e-507e7405141b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-74t5g" Jul 08 02:59:35.590184 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:35.590049 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8f8dbad-7d14-4946-a94e-507e7405141b-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-74t5g\" (UID: \"e8f8dbad-7d14-4946-a94e-507e7405141b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-74t5g" Jul 08 02:59:35.591202 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:35.591178 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8f8dbad-7d14-4946-a94e-507e7405141b-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-74t5g\" (UID: \"e8f8dbad-7d14-4946-a94e-507e7405141b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-74t5g" Jul 08 02:59:35.594631 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:35.594607 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8f8dbad-7d14-4946-a94e-507e7405141b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-74t5g\" (UID: \"e8f8dbad-7d14-4946-a94e-507e7405141b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-74t5g" Jul 08 02:59:35.595593 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:35.595567 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8f8dbad-7d14-4946-a94e-507e7405141b-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-74t5g\" (UID: \"e8f8dbad-7d14-4946-a94e-507e7405141b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-74t5g" Jul 08 02:59:35.597815 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:35.597794 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mbqx\" (UniqueName: \"kubernetes.io/projected/e8f8dbad-7d14-4946-a94e-507e7405141b-kube-api-access-6mbqx\") pod \"prometheus-operator-567f7b8b7c-74t5g\" (UID: \"e8f8dbad-7d14-4946-a94e-507e7405141b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-74t5g" Jul 08 02:59:35.641982 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:35.641945 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-74t5g" Jul 08 02:59:36.743639 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:36.743608 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:36.844141 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:36.844107 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:37.240936 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:37.240885 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:37.334476 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:37.334430 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:37.536737 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:37.536648 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:37.653424 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:37.653385 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:38.392872 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:38.392712 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-74t5g"] Jul 08 02:59:38.395842 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:59:38.395813 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f8dbad_7d14_4946_a94e_507e7405141b.slice/crio-267904b91a40931fe6d5b02ee560c1c813c8372d301788f3869c5935c355817c WatchSource:0}: Error finding container 267904b91a40931fe6d5b02ee560c1c813c8372d301788f3869c5935c355817c: Status 404 returned error can't find the container with id 267904b91a40931fe6d5b02ee560c1c813c8372d301788f3869c5935c355817c Jul 08 02:59:38.725148 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:38.725113 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:39.093388 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:39.093347 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rkrlf" event={"ID":"cd436540-823b-4c37-9a20-b8ebdf0937b7","Type":"ContainerStarted","Data":"9bc3959586b09cd2182e39ab70aabc109b8b06910f2542442a9358a147bc0d91"} Jul 08 02:59:39.093574 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:39.093413 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rkrlf" event={"ID":"cd436540-823b-4c37-9a20-b8ebdf0937b7","Type":"ContainerStarted","Data":"d5440ff58534a44805841e97bb3c03976858179858802796be5cf6f9b6c1135d"} Jul 08 02:59:39.093574 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:39.093475 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rkrlf" Jul 08 02:59:39.095676 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:39.095643 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ndv6v" event={"ID":"4bd7623c-f229-43b7-aefd-339ec594cf2c","Type":"ContainerStarted","Data":"e61480e314c989717116bba99c93a7b43da5f2a1343d6bd293906ee9839af20b"} Jul 08 02:59:39.097387 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:39.097356 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b49sr" event={"ID":"9489f6b7-c200-4e47-93f7-2f8a8e4be132","Type":"ContainerStarted","Data":"841754d6b93ce0570d2ae891d1224004a86c76b947d042e7affb89c66f5c1b2b"} Jul 08 02:59:39.098545 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:39.098518 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-74t5g" event={"ID":"e8f8dbad-7d14-4946-a94e-507e7405141b","Type":"ContainerStarted","Data":"267904b91a40931fe6d5b02ee560c1c813c8372d301788f3869c5935c355817c"} Jul 08 02:59:39.110130 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:39.110078 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rkrlf" podStartSLOduration=2.112771791 podStartE2EDuration="8.110058095s" totalImagesPullingTime="5.997286304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:39.109731212 +0000 UTC m=+53.767107085" watchObservedRunningTime="2026-07-08 02:59:39.110058095 +0000 UTC m=+53.767433970" Jul 08 02:59:39.125249 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:39.125195 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b49sr" podStartSLOduration=2.112973903 podStartE2EDuration="8.125180289s" totalImagesPullingTime="6.012206386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:39.124780374 +0000 UTC m=+53.782156245" watchObservedRunningTime="2026-07-08 02:59:39.125180289 +0000 UTC m=+53.782556159" Jul 08 02:59:39.730971 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:39.730937 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:39.831689 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:39.831645 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:40.070464 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:40.070425 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:40.106093 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:40.106051 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ndv6v" event={"ID":"4bd7623c-f229-43b7-aefd-339ec594cf2c","Type":"ContainerStarted","Data":"daf828b2b7d12df3348c6fe5f2e1a642c612aa9c02af74caefa016330a8240f9"} Jul 08 02:59:40.170898 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:40.170873 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:40.425391 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:40.425226 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:41.111901 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:41.111858 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-74t5g" event={"ID":"e8f8dbad-7d14-4946-a94e-507e7405141b","Type":"ContainerStarted","Data":"34f23bdf54e0dc8b22596ca775f98b0109ac8f1448b21ce02f80cb594ad3a4b8"} Jul 08 02:59:41.112325 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:41.111948 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-74t5g" event={"ID":"e8f8dbad-7d14-4946-a94e-507e7405141b","Type":"ContainerStarted","Data":"a8112da1a21fa77275c2bcbbd2e2d2a734ccab988997786d4c8217c66c5d6439"} Jul 08 02:59:41.129609 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:41.129551 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-74t5g" podStartSLOduration=4.373243025 podStartE2EDuration="6.129527565s" totalImagesPullingTime="1.75628454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:59:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:41.128933445 +0000 UTC m=+55.786309313" watchObservedRunningTime="2026-07-08 02:59:41.129527565 +0000 UTC m=+55.786903433" Jul 08 02:59:41.610062 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:41.610027 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:41.676283 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:41.676241 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:41.877453 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:41.877416 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:41.978437 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:41.978399 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:42.079149 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.079095 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:42.450894 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.450862 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:42.655667 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.655637 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:42.726982 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.726871 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7"] Jul 08 02:59:42.728871 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.728849 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-v2tk7"] Jul 08 02:59:42.729024 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.729002 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7" Jul 08 02:59:42.730615 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.730599 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:42.731747 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.731726 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 08 02:59:42.731847 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.731726 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 08 02:59:42.731847 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.731761 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-pf6sz\"" Jul 08 02:59:42.731967 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.731866 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 08 02:59:42.733383 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.733369 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 08 02:59:42.733565 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.733550 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 08 02:59:42.733643 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.733627 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 08 02:59:42.733697 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.733627 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9rm8r\"" Jul 08 02:59:42.745106 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.745074 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7"] Jul 08 02:59:42.756126 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.756101 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:42.834171 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.834131 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:42.848692 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.848656 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3c1a5c79-6201-494e-b04f-a3c4dbc7c90c-root\") pod \"node-exporter-v2tk7\" (UID: \"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c\") " pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:42.848807 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.848695 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c1a5c79-6201-494e-b04f-a3c4dbc7c90c-sys\") pod \"node-exporter-v2tk7\" (UID: \"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c\") " pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:42.848807 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.848714 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94fa01b1-87d6-41db-ad69-938f57cd77fe-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-zz8c7\" (UID: \"94fa01b1-87d6-41db-ad69-938f57cd77fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7" Jul 08 02:59:42.848807 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.848737 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/94fa01b1-87d6-41db-ad69-938f57cd77fe-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-zz8c7\" (UID: \"94fa01b1-87d6-41db-ad69-938f57cd77fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7" Jul 08 02:59:42.848807 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.848790 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3c1a5c79-6201-494e-b04f-a3c4dbc7c90c-node-exporter-wtmp\") pod \"node-exporter-v2tk7\" (UID: \"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c\") " pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:42.848981 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.848808 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c1a5c79-6201-494e-b04f-a3c4dbc7c90c-metrics-client-ca\") pod \"node-exporter-v2tk7\" (UID: \"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c\") " pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:42.848981 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.848833 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c1a5c79-6201-494e-b04f-a3c4dbc7c90c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v2tk7\" (UID: \"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c\") " pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:42.848981 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.848851 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94fa01b1-87d6-41db-ad69-938f57cd77fe-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-zz8c7\" (UID: \"94fa01b1-87d6-41db-ad69-938f57cd77fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7" Jul 08 02:59:42.848981 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.848867 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/94fa01b1-87d6-41db-ad69-938f57cd77fe-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-zz8c7\" (UID: \"94fa01b1-87d6-41db-ad69-938f57cd77fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7" Jul 08 02:59:42.848981 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.848912 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3c1a5c79-6201-494e-b04f-a3c4dbc7c90c-node-exporter-textfile\") pod \"node-exporter-v2tk7\" (UID: \"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c\") " pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:42.848981 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.848969 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3c1a5c79-6201-494e-b04f-a3c4dbc7c90c-node-exporter-tls\") pod \"node-exporter-v2tk7\" (UID: \"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c\") " pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:42.849182 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.849027 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3c1a5c79-6201-494e-b04f-a3c4dbc7c90c-node-exporter-accelerators-collector-config\") pod \"node-exporter-v2tk7\" (UID: \"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c\") " pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:42.849182 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.849057 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk6tw\" (UniqueName: \"kubernetes.io/projected/3c1a5c79-6201-494e-b04f-a3c4dbc7c90c-kube-api-access-zk6tw\") pod \"node-exporter-v2tk7\" (UID: \"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c\") " pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:42.849182 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.849092 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/94fa01b1-87d6-41db-ad69-938f57cd77fe-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-zz8c7\" (UID: \"94fa01b1-87d6-41db-ad69-938f57cd77fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7" Jul 08 02:59:42.849182 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.849109 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzcr6\" (UniqueName: \"kubernetes.io/projected/94fa01b1-87d6-41db-ad69-938f57cd77fe-kube-api-access-xzcr6\") pod \"kube-state-metrics-7c9dbb8b7d-zz8c7\" (UID: \"94fa01b1-87d6-41db-ad69-938f57cd77fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7" Jul 08 02:59:42.900735 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.900698 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:42.950170 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.950133 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3c1a5c79-6201-494e-b04f-a3c4dbc7c90c-node-exporter-accelerators-collector-config\") pod \"node-exporter-v2tk7\" (UID: \"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c\") " pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:42.950170 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.950173 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk6tw\" (UniqueName: \"kubernetes.io/projected/3c1a5c79-6201-494e-b04f-a3c4dbc7c90c-kube-api-access-zk6tw\") pod \"node-exporter-v2tk7\" (UID: \"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c\") " pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:42.950428 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.950201 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/94fa01b1-87d6-41db-ad69-938f57cd77fe-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-zz8c7\" (UID: \"94fa01b1-87d6-41db-ad69-938f57cd77fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7" Jul 08 02:59:42.950428 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.950223 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzcr6\" (UniqueName: \"kubernetes.io/projected/94fa01b1-87d6-41db-ad69-938f57cd77fe-kube-api-access-xzcr6\") pod \"kube-state-metrics-7c9dbb8b7d-zz8c7\" (UID: \"94fa01b1-87d6-41db-ad69-938f57cd77fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7" Jul 08 02:59:42.950428 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.950276 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3c1a5c79-6201-494e-b04f-a3c4dbc7c90c-root\") pod \"node-exporter-v2tk7\" (UID: \"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c\") " pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:42.950428 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.950301 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c1a5c79-6201-494e-b04f-a3c4dbc7c90c-sys\") pod \"node-exporter-v2tk7\" (UID: \"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c\") " pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:42.950428 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.950324 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94fa01b1-87d6-41db-ad69-938f57cd77fe-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-zz8c7\" (UID: \"94fa01b1-87d6-41db-ad69-938f57cd77fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7" Jul 08 02:59:42.950428 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.950346 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/94fa01b1-87d6-41db-ad69-938f57cd77fe-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-zz8c7\" (UID: \"94fa01b1-87d6-41db-ad69-938f57cd77fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7" Jul 08 02:59:42.950428 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.950375 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3c1a5c79-6201-494e-b04f-a3c4dbc7c90c-node-exporter-wtmp\") pod \"node-exporter-v2tk7\" (UID: \"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c\") " pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:42.950428 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.950379 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3c1a5c79-6201-494e-b04f-a3c4dbc7c90c-root\") pod \"node-exporter-v2tk7\" (UID: \"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c\") " pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:42.950428 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.950402 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c1a5c79-6201-494e-b04f-a3c4dbc7c90c-metrics-client-ca\") pod \"node-exporter-v2tk7\" (UID: \"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c\") " pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:42.950428 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.950382 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c1a5c79-6201-494e-b04f-a3c4dbc7c90c-sys\") pod \"node-exporter-v2tk7\" (UID: \"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c\") " pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:42.950825 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.950450 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c1a5c79-6201-494e-b04f-a3c4dbc7c90c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v2tk7\" (UID: \"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c\") " pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:42.950825 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.950485 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94fa01b1-87d6-41db-ad69-938f57cd77fe-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-zz8c7\" (UID: \"94fa01b1-87d6-41db-ad69-938f57cd77fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7" Jul 08 02:59:42.950825 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:42.950497 2564 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 08 02:59:42.950825 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.950516 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/94fa01b1-87d6-41db-ad69-938f57cd77fe-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-zz8c7\" (UID: \"94fa01b1-87d6-41db-ad69-938f57cd77fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7" Jul 08 02:59:42.950825 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.950558 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3c1a5c79-6201-494e-b04f-a3c4dbc7c90c-node-exporter-wtmp\") pod \"node-exporter-v2tk7\" (UID: \"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c\") " pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:42.950825 ip-10-0-128-70 kubenswrapper[2564]: E0708 02:59:42.950579 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94fa01b1-87d6-41db-ad69-938f57cd77fe-kube-state-metrics-tls podName:94fa01b1-87d6-41db-ad69-938f57cd77fe nodeName:}" failed. No retries permitted until 2026-07-08 02:59:43.450558413 +0000 UTC m=+58.107934282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/94fa01b1-87d6-41db-ad69-938f57cd77fe-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-zz8c7" (UID: "94fa01b1-87d6-41db-ad69-938f57cd77fe") : secret "kube-state-metrics-tls" not found Jul 08 02:59:42.950825 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.950685 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3c1a5c79-6201-494e-b04f-a3c4dbc7c90c-node-exporter-textfile\") pod \"node-exporter-v2tk7\" (UID: \"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c\") " pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:42.950825 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.950746 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3c1a5c79-6201-494e-b04f-a3c4dbc7c90c-node-exporter-tls\") pod \"node-exporter-v2tk7\" (UID: \"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c\") " pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:42.951276 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.951178 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/94fa01b1-87d6-41db-ad69-938f57cd77fe-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-zz8c7\" (UID: \"94fa01b1-87d6-41db-ad69-938f57cd77fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7" Jul 08 02:59:42.951340 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.951321 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3c1a5c79-6201-494e-b04f-a3c4dbc7c90c-node-exporter-textfile\") pod \"node-exporter-v2tk7\" (UID: \"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c\") " pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:42.951391 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.951342 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/94fa01b1-87d6-41db-ad69-938f57cd77fe-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-zz8c7\" (UID: \"94fa01b1-87d6-41db-ad69-938f57cd77fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7" Jul 08 02:59:42.951540 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.951516 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3c1a5c79-6201-494e-b04f-a3c4dbc7c90c-node-exporter-accelerators-collector-config\") pod \"node-exporter-v2tk7\" (UID: \"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c\") " pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:42.951595 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.951516 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/94fa01b1-87d6-41db-ad69-938f57cd77fe-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-zz8c7\" (UID: \"94fa01b1-87d6-41db-ad69-938f57cd77fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7" Jul 08 02:59:42.951693 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.951664 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c1a5c79-6201-494e-b04f-a3c4dbc7c90c-metrics-client-ca\") pod \"node-exporter-v2tk7\" (UID: \"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c\") " pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:42.954658 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.954634 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3c1a5c79-6201-494e-b04f-a3c4dbc7c90c-node-exporter-tls\") pod \"node-exporter-v2tk7\" (UID: \"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c\") " pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:42.954658 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.954639 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c1a5c79-6201-494e-b04f-a3c4dbc7c90c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v2tk7\" (UID: \"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c\") " pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:42.954815 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.954682 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/94fa01b1-87d6-41db-ad69-938f57cd77fe-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-zz8c7\" (UID: \"94fa01b1-87d6-41db-ad69-938f57cd77fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7" Jul 08 02:59:42.960816 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.960792 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk6tw\" (UniqueName: \"kubernetes.io/projected/3c1a5c79-6201-494e-b04f-a3c4dbc7c90c-kube-api-access-zk6tw\") pod \"node-exporter-v2tk7\" (UID: \"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c\") " pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:42.960954 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:42.960798 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzcr6\" (UniqueName: \"kubernetes.io/projected/94fa01b1-87d6-41db-ad69-938f57cd77fe-kube-api-access-xzcr6\") pod \"kube-state-metrics-7c9dbb8b7d-zz8c7\" (UID: \"94fa01b1-87d6-41db-ad69-938f57cd77fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7" Jul 08 02:59:43.042632 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.042552 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:43.048484 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.048461 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v2tk7" Jul 08 02:59:43.058328 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:59:43.058297 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c1a5c79_6201_494e_b04f_a3c4dbc7c90c.slice/crio-6965bc5ef3ed93b2e4233f3e7c30cbd49eac2b1b08b578a80a1cda8bcece294c WatchSource:0}: Error finding container 6965bc5ef3ed93b2e4233f3e7c30cbd49eac2b1b08b578a80a1cda8bcece294c: Status 404 returned error can't find the container with id 6965bc5ef3ed93b2e4233f3e7c30cbd49eac2b1b08b578a80a1cda8bcece294c Jul 08 02:59:43.124300 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.124267 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v2tk7" event={"ID":"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c","Type":"ContainerStarted","Data":"6965bc5ef3ed93b2e4233f3e7c30cbd49eac2b1b08b578a80a1cda8bcece294c"} Jul 08 02:59:43.126289 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.126260 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ndv6v" event={"ID":"4bd7623c-f229-43b7-aefd-339ec594cf2c","Type":"ContainerStarted","Data":"835d1f41e87441899fae09b5070f057d58246cf10a4428f7f70d6952d08b1e0c"} Jul 08 02:59:43.145850 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.145810 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:43.305266 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.305175 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:43.454786 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.454744 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/94fa01b1-87d6-41db-ad69-938f57cd77fe-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-zz8c7\" (UID: \"94fa01b1-87d6-41db-ad69-938f57cd77fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7" Jul 08 02:59:43.457221 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.457204 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/94fa01b1-87d6-41db-ad69-938f57cd77fe-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-zz8c7\" (UID: \"94fa01b1-87d6-41db-ad69-938f57cd77fe\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7" Jul 08 02:59:43.642320 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.642289 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7" Jul 08 02:59:43.668621 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.668590 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:43.751747 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.751681 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ndv6v" podStartSLOduration=2.594695467 podStartE2EDuration="12.751655935s" totalImagesPullingTime="10.156960468s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:59:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:43.144359237 +0000 UTC m=+57.801735109" watchObservedRunningTime="2026-07-08 02:59:43.751655935 +0000 UTC m=+58.409031789" Jul 08 02:59:43.752487 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.752464 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 02:59:43.754576 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.754475 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:43.758910 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.755716 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.759688 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.759309 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xdpxj\"" Jul 08 02:59:43.759688 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.759372 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 08 02:59:43.759688 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.759469 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 08 02:59:43.761954 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.761580 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 08 02:59:43.761954 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.761594 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 08 02:59:43.761954 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.761599 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 08 02:59:43.761954 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.761827 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 08 02:59:43.762201 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.762036 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 08 02:59:43.762201 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.762039 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 08 02:59:43.762201 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.762039 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 08 02:59:43.771964 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.771775 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 02:59:43.798277 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.798157 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7"] Jul 08 02:59:43.803162 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:59:43.803122 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94fa01b1_87d6_41db_ad69_938f57cd77fe.slice/crio-6533eb61bf97ff4bd16bb82a927b63ed1a4a48415da8ca60f5e50d01b2607622 WatchSource:0}: Error finding container 6533eb61bf97ff4bd16bb82a927b63ed1a4a48415da8ca60f5e50d01b2607622: Status 404 returned error can't find the container with id 6533eb61bf97ff4bd16bb82a927b63ed1a4a48415da8ca60f5e50d01b2607622 Jul 08 02:59:43.859690 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.859656 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-config-volume\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.859857 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.859696 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.859857 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.859728 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.859857 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.859837 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/de0f3a7e-a05c-4274-bc1e-657236a9bfea-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.860013 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.859882 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de0f3a7e-a05c-4274-bc1e-657236a9bfea-config-out\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.860013 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.859913 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.860013 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.859957 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de0f3a7e-a05c-4274-bc1e-657236a9bfea-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.860013 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.859996 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-web-config\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.860132 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.860026 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de0f3a7e-a05c-4274-bc1e-657236a9bfea-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.860132 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.860063 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.860212 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.860141 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de0f3a7e-a05c-4274-bc1e-657236a9bfea-tls-assets\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.860212 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.860180 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.860212 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.860206 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bltlt\" (UniqueName: \"kubernetes.io/projected/de0f3a7e-a05c-4274-bc1e-657236a9bfea-kube-api-access-bltlt\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.869355 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.869313 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:43.961372 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.961299 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de0f3a7e-a05c-4274-bc1e-657236a9bfea-tls-assets\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.961372 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.961344 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.961372 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.961366 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bltlt\" (UniqueName: \"kubernetes.io/projected/de0f3a7e-a05c-4274-bc1e-657236a9bfea-kube-api-access-bltlt\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.961624 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.961388 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-config-volume\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.961624 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.961510 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.961624 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.961553 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.961624 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.961599 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/de0f3a7e-a05c-4274-bc1e-657236a9bfea-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.961812 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.961642 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de0f3a7e-a05c-4274-bc1e-657236a9bfea-config-out\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.961812 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.961671 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.961812 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.961696 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de0f3a7e-a05c-4274-bc1e-657236a9bfea-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.961812 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.961728 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-web-config\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.961812 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.961759 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de0f3a7e-a05c-4274-bc1e-657236a9bfea-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.961812 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.961797 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.963718 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.963330 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de0f3a7e-a05c-4274-bc1e-657236a9bfea-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.964489 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.964460 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de0f3a7e-a05c-4274-bc1e-657236a9bfea-config-out\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.964715 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.964694 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-config-volume\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.964876 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.964850 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de0f3a7e-a05c-4274-bc1e-657236a9bfea-tls-assets\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.964974 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.964874 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.964974 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.964956 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/de0f3a7e-a05c-4274-bc1e-657236a9bfea-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.965096 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.965038 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.965721 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.965682 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de0f3a7e-a05c-4274-bc1e-657236a9bfea-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.965897 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.965876 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.966817 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.966797 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.966872 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.966801 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.967240 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.967222 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-web-config\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:43.971061 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:43.971031 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bltlt\" (UniqueName: \"kubernetes.io/projected/de0f3a7e-a05c-4274-bc1e-657236a9bfea-kube-api-access-bltlt\") pod \"alertmanager-main-0\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:44.070173 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.070141 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 02:59:44.130965 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.130902 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7" event={"ID":"94fa01b1-87d6-41db-ad69-938f57cd77fe","Type":"ContainerStarted","Data":"6533eb61bf97ff4bd16bb82a927b63ed1a4a48415da8ca60f5e50d01b2607622"} Jul 08 02:59:44.300474 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.300440 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 02:59:44.306787 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:59:44.306755 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde0f3a7e_a05c_4274_bc1e_657236a9bfea.slice/crio-51ef688c1d0b2d110f7dfb2c3efbcbf524f0ba2113a0f1cc57c50f2662c1d10c WatchSource:0}: Error finding container 51ef688c1d0b2d110f7dfb2c3efbcbf524f0ba2113a0f1cc57c50f2662c1d10c: Status 404 returned error can't find the container with id 51ef688c1d0b2d110f7dfb2c3efbcbf524f0ba2113a0f1cc57c50f2662c1d10c Jul 08 02:59:44.600024 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.599941 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:44.671138 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.671100 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-575dbdcf76-mt7x6"] Jul 08 02:59:44.674690 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.673971 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" Jul 08 02:59:44.677032 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.677005 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7csscg83qln0c\"" Jul 08 02:59:44.677161 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.677044 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 08 02:59:44.677161 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.677077 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rs2vq\"" Jul 08 02:59:44.677161 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.677117 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 08 02:59:44.677300 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.677005 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 08 02:59:44.677300 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.677166 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 08 02:59:44.677300 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.677215 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 08 02:59:44.692293 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.692263 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-575dbdcf76-mt7x6"] Jul 08 02:59:44.697101 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.697078 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:44.871013 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.870903 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2701c118-eb37-4c11-917c-010285e54cb7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-575dbdcf76-mt7x6\" (UID: \"2701c118-eb37-4c11-917c-010285e54cb7\") " pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" Jul 08 02:59:44.871013 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.870986 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2701c118-eb37-4c11-917c-010285e54cb7-metrics-client-ca\") pod \"thanos-querier-575dbdcf76-mt7x6\" (UID: \"2701c118-eb37-4c11-917c-010285e54cb7\") " pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" Jul 08 02:59:44.871247 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.871105 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2701c118-eb37-4c11-917c-010285e54cb7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-575dbdcf76-mt7x6\" (UID: \"2701c118-eb37-4c11-917c-010285e54cb7\") " pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" Jul 08 02:59:44.871247 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.871143 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2701c118-eb37-4c11-917c-010285e54cb7-secret-thanos-querier-tls\") pod \"thanos-querier-575dbdcf76-mt7x6\" (UID: \"2701c118-eb37-4c11-917c-010285e54cb7\") " pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" Jul 08 02:59:44.871247 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.871170 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2701c118-eb37-4c11-917c-010285e54cb7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-575dbdcf76-mt7x6\" (UID: \"2701c118-eb37-4c11-917c-010285e54cb7\") " pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" Jul 08 02:59:44.871247 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.871198 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2701c118-eb37-4c11-917c-010285e54cb7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-575dbdcf76-mt7x6\" (UID: \"2701c118-eb37-4c11-917c-010285e54cb7\") " pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" Jul 08 02:59:44.871247 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.871241 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2701c118-eb37-4c11-917c-010285e54cb7-secret-grpc-tls\") pod \"thanos-querier-575dbdcf76-mt7x6\" (UID: \"2701c118-eb37-4c11-917c-010285e54cb7\") " pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" Jul 08 02:59:44.871485 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.871332 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn2c9\" (UniqueName: \"kubernetes.io/projected/2701c118-eb37-4c11-917c-010285e54cb7-kube-api-access-wn2c9\") pod \"thanos-querier-575dbdcf76-mt7x6\" (UID: \"2701c118-eb37-4c11-917c-010285e54cb7\") " pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" Jul 08 02:59:44.972153 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.972121 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn2c9\" (UniqueName: \"kubernetes.io/projected/2701c118-eb37-4c11-917c-010285e54cb7-kube-api-access-wn2c9\") pod \"thanos-querier-575dbdcf76-mt7x6\" (UID: \"2701c118-eb37-4c11-917c-010285e54cb7\") " pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" Jul 08 02:59:44.972352 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.972181 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2701c118-eb37-4c11-917c-010285e54cb7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-575dbdcf76-mt7x6\" (UID: \"2701c118-eb37-4c11-917c-010285e54cb7\") " pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" Jul 08 02:59:44.972352 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.972216 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2701c118-eb37-4c11-917c-010285e54cb7-metrics-client-ca\") pod \"thanos-querier-575dbdcf76-mt7x6\" (UID: \"2701c118-eb37-4c11-917c-010285e54cb7\") " pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" Jul 08 02:59:44.972352 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.972280 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2701c118-eb37-4c11-917c-010285e54cb7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-575dbdcf76-mt7x6\" (UID: \"2701c118-eb37-4c11-917c-010285e54cb7\") " pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" Jul 08 02:59:44.972352 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.972305 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2701c118-eb37-4c11-917c-010285e54cb7-secret-thanos-querier-tls\") pod \"thanos-querier-575dbdcf76-mt7x6\" (UID: \"2701c118-eb37-4c11-917c-010285e54cb7\") " pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" Jul 08 02:59:44.972352 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.972322 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2701c118-eb37-4c11-917c-010285e54cb7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-575dbdcf76-mt7x6\" (UID: \"2701c118-eb37-4c11-917c-010285e54cb7\") " pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" Jul 08 02:59:44.972352 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.972345 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2701c118-eb37-4c11-917c-010285e54cb7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-575dbdcf76-mt7x6\" (UID: \"2701c118-eb37-4c11-917c-010285e54cb7\") " pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" Jul 08 02:59:44.972640 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.972375 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2701c118-eb37-4c11-917c-010285e54cb7-secret-grpc-tls\") pod \"thanos-querier-575dbdcf76-mt7x6\" (UID: \"2701c118-eb37-4c11-917c-010285e54cb7\") " pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" Jul 08 02:59:44.973797 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.973746 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2701c118-eb37-4c11-917c-010285e54cb7-metrics-client-ca\") pod \"thanos-querier-575dbdcf76-mt7x6\" (UID: \"2701c118-eb37-4c11-917c-010285e54cb7\") " pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" Jul 08 02:59:44.975991 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.975942 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2701c118-eb37-4c11-917c-010285e54cb7-secret-grpc-tls\") pod \"thanos-querier-575dbdcf76-mt7x6\" (UID: \"2701c118-eb37-4c11-917c-010285e54cb7\") " pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" Jul 08 02:59:44.976189 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.976162 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2701c118-eb37-4c11-917c-010285e54cb7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-575dbdcf76-mt7x6\" (UID: \"2701c118-eb37-4c11-917c-010285e54cb7\") " pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" Jul 08 02:59:44.976887 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.976840 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2701c118-eb37-4c11-917c-010285e54cb7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-575dbdcf76-mt7x6\" (UID: \"2701c118-eb37-4c11-917c-010285e54cb7\") " pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" Jul 08 02:59:44.977698 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.977670 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2701c118-eb37-4c11-917c-010285e54cb7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-575dbdcf76-mt7x6\" (UID: \"2701c118-eb37-4c11-917c-010285e54cb7\") " pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" Jul 08 02:59:44.977863 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.977803 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2701c118-eb37-4c11-917c-010285e54cb7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-575dbdcf76-mt7x6\" (UID: \"2701c118-eb37-4c11-917c-010285e54cb7\") " pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" Jul 08 02:59:44.978121 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.978081 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2701c118-eb37-4c11-917c-010285e54cb7-secret-thanos-querier-tls\") pod \"thanos-querier-575dbdcf76-mt7x6\" (UID: \"2701c118-eb37-4c11-917c-010285e54cb7\") " pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" Jul 08 02:59:44.981525 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.981505 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn2c9\" (UniqueName: \"kubernetes.io/projected/2701c118-eb37-4c11-917c-010285e54cb7-kube-api-access-wn2c9\") pod \"thanos-querier-575dbdcf76-mt7x6\" (UID: \"2701c118-eb37-4c11-917c-010285e54cb7\") " pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" Jul 08 02:59:44.985457 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:44.985435 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" Jul 08 02:59:45.136035 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:45.135999 2564 generic.go:356] "Generic (PLEG): container finished" podID="3c1a5c79-6201-494e-b04f-a3c4dbc7c90c" containerID="2b100302489770284917798a5c8d3441a2a15b02d7a554ecafca8fb9443a3ff7" exitCode=0 Jul 08 02:59:45.136228 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:45.136075 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v2tk7" event={"ID":"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c","Type":"ContainerDied","Data":"2b100302489770284917798a5c8d3441a2a15b02d7a554ecafca8fb9443a3ff7"} Jul 08 02:59:45.138027 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:45.138003 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de0f3a7e-a05c-4274-bc1e-657236a9bfea","Type":"ContainerStarted","Data":"51ef688c1d0b2d110f7dfb2c3efbcbf524f0ba2113a0f1cc57c50f2662c1d10c"} Jul 08 02:59:45.550737 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:45.550691 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-575dbdcf76-mt7x6"] Jul 08 02:59:45.564561 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:59:45.564495 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2701c118_eb37_4c11_917c_010285e54cb7.slice/crio-238217a844683070aa07c9abd29b9154bbb4d88c99172e065cc1d5ee70bfc6dc WatchSource:0}: Error finding container 238217a844683070aa07c9abd29b9154bbb4d88c99172e065cc1d5ee70bfc6dc: Status 404 returned error can't find the container with id 238217a844683070aa07c9abd29b9154bbb4d88c99172e065cc1d5ee70bfc6dc Jul 08 02:59:45.679161 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:45.679016 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:45.732308 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:45.732269 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:45.786306 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:45.786279 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:46.148736 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:46.148686 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7" event={"ID":"94fa01b1-87d6-41db-ad69-938f57cd77fe","Type":"ContainerStarted","Data":"48f5e26b647a56060047e15507074d4609586464bd4a952cd70091e580aa14b8"} Jul 08 02:59:46.148947 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:46.148803 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7" event={"ID":"94fa01b1-87d6-41db-ad69-938f57cd77fe","Type":"ContainerStarted","Data":"b1b90aea4b168d76932d09326665146dffe64f69132f2e77c1d0d486707fed74"} Jul 08 02:59:46.148947 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:46.148833 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7" event={"ID":"94fa01b1-87d6-41db-ad69-938f57cd77fe","Type":"ContainerStarted","Data":"774d3a4f1b7df79b7ee66ad87a8f741a0017ce996500503c955762e10b3cec8d"} Jul 08 02:59:46.150269 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:46.150204 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" event={"ID":"2701c118-eb37-4c11-917c-010285e54cb7","Type":"ContainerStarted","Data":"238217a844683070aa07c9abd29b9154bbb4d88c99172e065cc1d5ee70bfc6dc"} Jul 08 02:59:46.152551 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:46.152522 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v2tk7" event={"ID":"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c","Type":"ContainerStarted","Data":"40beb4dedd5590a83bcc09f1eaedd58b0f5662e4375159946fc90b59f5bc850e"} Jul 08 02:59:46.152659 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:46.152571 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v2tk7" event={"ID":"3c1a5c79-6201-494e-b04f-a3c4dbc7c90c","Type":"ContainerStarted","Data":"aadab0c6db87f56e7b1bd8ddf8df4ac6d7bf80b2aaeb59ad8d54946b250bd37b"} Jul 08 02:59:46.154359 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:46.154334 2564 generic.go:356] "Generic (PLEG): container finished" podID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerID="8c29c8f036afd01a5fbd4388075ac0a1380da38a04f6e2898559b9b32cd2c280" exitCode=0 Jul 08 02:59:46.154451 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:46.154385 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de0f3a7e-a05c-4274-bc1e-657236a9bfea","Type":"ContainerDied","Data":"8c29c8f036afd01a5fbd4388075ac0a1380da38a04f6e2898559b9b32cd2c280"} Jul 08 02:59:46.169605 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:46.169555 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-zz8c7" podStartSLOduration=2.598528682 podStartE2EDuration="4.169538117s" totalImagesPullingTime="1.571009435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:59:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:46.16874857 +0000 UTC m=+60.826124443" watchObservedRunningTime="2026-07-08 02:59:46.169538117 +0000 UTC m=+60.826914005" Jul 08 02:59:46.190241 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:46.190191 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-v2tk7" podStartSLOduration=2.979705644 podStartE2EDuration="4.190175963s" totalImagesPullingTime="1.131222392s" totalInitContainerRuntime="79.247927ms" isStatefulPod=true podCreationTimestamp="2026-07-08 02:59:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:46.189406756 +0000 UTC m=+60.846782628" watchObservedRunningTime="2026-07-08 02:59:46.190175963 +0000 UTC m=+60.847551834" Jul 08 02:59:47.034711 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:47.034677 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lv2jb" Jul 08 02:59:47.170566 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:47.170525 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:47.253294 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:47.253222 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:47.460415 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:47.460378 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:47.552387 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:47.552346 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:47.749884 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:47.749807 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:47.816671 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:47.816642 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:48.165376 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:48.165344 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" event={"ID":"2701c118-eb37-4c11-917c-010285e54cb7","Type":"ContainerStarted","Data":"3e92c412d49b1c94263e257966ffae0d0f6f72711c6fcbcf0f906b952573b431"} Jul 08 02:59:48.167596 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:48.167572 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de0f3a7e-a05c-4274-bc1e-657236a9bfea","Type":"ContainerStarted","Data":"db79e07a64fd78f2dea8cfbbe1355a31318404665dd5ce17bc41afe5c2e295f1"} Jul 08 02:59:48.975901 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:48.975866 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:48.980795 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:48.980764 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 02:59:48.989698 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:48.989662 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:48.992317 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:48.992293 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 08 02:59:48.992564 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:48.992548 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 08 02:59:48.992627 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:48.992594 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 08 02:59:48.992678 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:48.992651 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 08 02:59:48.992678 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:48.992667 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 08 02:59:48.994103 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:48.993423 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bkaojbjkcl3af\"" Jul 08 02:59:48.994103 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:48.993471 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tk22q\"" Jul 08 02:59:48.994103 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:48.993506 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 08 02:59:48.994103 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:48.993722 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 08 02:59:48.994103 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:48.993842 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 08 02:59:48.994103 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:48.993978 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 08 02:59:48.994103 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:48.993994 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 08 02:59:48.994495 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:48.994221 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 08 02:59:48.994495 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:48.994355 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 08 02:59:48.994495 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:48.994466 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 08 02:59:48.994597 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:48.994566 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 08 02:59:48.998655 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:48.998622 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 08 02:59:49.002359 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.002337 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 02:59:49.010034 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.010011 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.010155 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.010054 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5141e7b4-0262-4a1d-809f-752a82e5f41f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.010155 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.010074 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.010155 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.010143 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.010305 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.010185 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5141e7b4-0262-4a1d-809f-752a82e5f41f-config-out\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.010305 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.010209 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.010305 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.010236 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.010305 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.010285 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sz6s\" (UniqueName: \"kubernetes.io/projected/5141e7b4-0262-4a1d-809f-752a82e5f41f-kube-api-access-2sz6s\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.010428 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.010334 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.010428 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.010385 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.010428 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.010409 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.010536 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.010434 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.010536 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.010463 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.010536 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.010495 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.010659 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.010559 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-config\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.010659 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.010595 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.010659 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.010642 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.010786 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.010671 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.010786 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.010700 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-web-config\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.010786 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.010754 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.108833 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.108808 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rkrlf" Jul 08 02:59:49.111397 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.111373 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5141e7b4-0262-4a1d-809f-752a82e5f41f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.111577 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.111561 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.111724 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.111708 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.111862 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.111847 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5141e7b4-0262-4a1d-809f-752a82e5f41f-config-out\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.112001 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.111984 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.112156 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.112101 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.112288 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.112271 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sz6s\" (UniqueName: \"kubernetes.io/projected/5141e7b4-0262-4a1d-809f-752a82e5f41f-kube-api-access-2sz6s\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.112409 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.112393 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.112516 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.112501 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.112599 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.112572 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.112686 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.112671 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.112779 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.112765 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.114509 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.112908 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.114509 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.112940 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.114509 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.112990 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.114509 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.113002 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.114509 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.112432 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.114509 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.113061 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-config\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.114509 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.113146 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.114509 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.113213 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.114509 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.113247 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.114509 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.113276 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-web-config\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.114509 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.113327 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.114509 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.113408 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.114509 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.113621 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.114509 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.114081 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.115266 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.114553 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.115266 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.114552 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.115266 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.114707 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5141e7b4-0262-4a1d-809f-752a82e5f41f-config-out\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.115871 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.115815 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5141e7b4-0262-4a1d-809f-752a82e5f41f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.115986 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.115956 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.117987 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.117677 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.118294 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.118235 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.118680 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.118639 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.118680 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.118666 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-web-config\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.118817 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.118713 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.118817 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.118741 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.119429 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.119387 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-config\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.122238 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.122215 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.123717 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.123694 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sz6s\" (UniqueName: \"kubernetes.io/projected/5141e7b4-0262-4a1d-809f-752a82e5f41f-kube-api-access-2sz6s\") pod \"prometheus-k8s-0\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.176293 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.176259 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de0f3a7e-a05c-4274-bc1e-657236a9bfea","Type":"ContainerStarted","Data":"75630a037d2f93a7d44ca42f357ce1c71ab9127cfe7c4b813e1e7a0c7d8b2279"} Jul 08 02:59:49.176699 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.176359 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de0f3a7e-a05c-4274-bc1e-657236a9bfea","Type":"ContainerStarted","Data":"8a302955e9a7306c2a963a73d03dd19e0562c53f282166fcc80e2a4fc746242f"} Jul 08 02:59:49.176699 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.176404 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de0f3a7e-a05c-4274-bc1e-657236a9bfea","Type":"ContainerStarted","Data":"3b992808db3fc67b03da8df1b5c23ce48ca2a2981795934c64cc8c62bfbbb550"} Jul 08 02:59:49.176699 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.176444 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de0f3a7e-a05c-4274-bc1e-657236a9bfea","Type":"ContainerStarted","Data":"0798b0ea7d27cfdeb45da811020926dff3c668bdfa26bcdbdcdd43721645695a"} Jul 08 02:59:49.176699 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.176489 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de0f3a7e-a05c-4274-bc1e-657236a9bfea","Type":"ContainerStarted","Data":"1e70432784c57e12889d55e1823c8df41b29d634e355e94fce506f1935dec2e9"} Jul 08 02:59:49.179088 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.179064 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" event={"ID":"2701c118-eb37-4c11-917c-010285e54cb7","Type":"ContainerStarted","Data":"ee2f2f28e5c70f7da6f4b17adc8abd6f6b3f9bda2f57f51fbdd17e69b44426c3"} Jul 08 02:59:49.179207 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.179119 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" event={"ID":"2701c118-eb37-4c11-917c-010285e54cb7","Type":"ContainerStarted","Data":"efe16b95724fea06fd3a66db5d0a0dc57bbccec019b433145049f260c85b05bc"} Jul 08 02:59:49.179207 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.179146 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" event={"ID":"2701c118-eb37-4c11-917c-010285e54cb7","Type":"ContainerStarted","Data":"1485422ab98790d05922cdaf8168a70357f96440e888d3acf668842b294c9890"} Jul 08 02:59:49.179207 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.179168 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" event={"ID":"2701c118-eb37-4c11-917c-010285e54cb7","Type":"ContainerStarted","Data":"09fa0ea532349bda9f34ad1e8d441c33e2a02c2bdfa033359e5cff5214ba4672"} Jul 08 02:59:49.206595 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.205762 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.5520259100000002 podStartE2EDuration="6.205743316s" totalImagesPullingTime="3.630102499s" totalInitContainerRuntime="23.614907ms" isStatefulPod=false podCreationTimestamp="2026-07-08 02:59:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:49.204993496 +0000 UTC m=+63.862369366" watchObservedRunningTime="2026-07-08 02:59:49.205743316 +0000 UTC m=+63.863119189" Jul 08 02:59:49.303522 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.303479 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:49.458155 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.458109 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 02:59:49.465101 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:59:49.465073 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5141e7b4_0262_4a1d_809f_752a82e5f41f.slice/crio-43d9016c8a7a7a3440cf9c5dc639232076f946df89fe578f31c9a2c905e7887b WatchSource:0}: Error finding container 43d9016c8a7a7a3440cf9c5dc639232076f946df89fe578f31c9a2c905e7887b: Status 404 returned error can't find the container with id 43d9016c8a7a7a3440cf9c5dc639232076f946df89fe578f31c9a2c905e7887b Jul 08 02:59:49.963868 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:49.963826 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:50.065568 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:50.065531 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:50.185268 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:50.185223 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" event={"ID":"2701c118-eb37-4c11-917c-010285e54cb7","Type":"ContainerStarted","Data":"d63de135d36767d48552702c37dd500b0262a674daf11ec57c13442040ab4157"} Jul 08 02:59:50.185679 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:50.185371 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" Jul 08 02:59:50.187172 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:50.187149 2564 generic.go:356] "Generic (PLEG): container finished" podID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerID="731bb7c7f40b76b5b5a6e519787af4971c6fd088111ad0fb687a77dbe9cc6b81" exitCode=0 Jul 08 02:59:50.187303 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:50.187228 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5141e7b4-0262-4a1d-809f-752a82e5f41f","Type":"ContainerDied","Data":"731bb7c7f40b76b5b5a6e519787af4971c6fd088111ad0fb687a77dbe9cc6b81"} Jul 08 02:59:50.187303 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:50.187286 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5141e7b4-0262-4a1d-809f-752a82e5f41f","Type":"ContainerStarted","Data":"43d9016c8a7a7a3440cf9c5dc639232076f946df89fe578f31c9a2c905e7887b"} Jul 08 02:59:50.212653 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:50.212437 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" podStartSLOduration=2.955620407 podStartE2EDuration="6.212417604s" totalImagesPullingTime="3.256797197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:59:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:50.210689573 +0000 UTC m=+64.868065434" watchObservedRunningTime="2026-07-08 02:59:50.212417604 +0000 UTC m=+64.869793475" Jul 08 02:59:51.538904 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:51.538860 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f5283b1-208f-415c-819f-c3e479102863-metrics-certs\") pod \"network-metrics-daemon-lsmbz\" (UID: \"8f5283b1-208f-415c-819f-c3e479102863\") " pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:59:51.541395 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:51.541375 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 02:59:51.552232 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:51.552197 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f5283b1-208f-415c-819f-c3e479102863-metrics-certs\") pod \"network-metrics-daemon-lsmbz\" (UID: \"8f5283b1-208f-415c-819f-c3e479102863\") " pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:59:51.639661 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:51.639618 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh64r\" (UniqueName: \"kubernetes.io/projected/36d7c073-6c29-456f-a540-34e61812ac39-kube-api-access-nh64r\") pod \"network-check-target-hldxf\" (UID: \"36d7c073-6c29-456f-a540-34e61812ac39\") " pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:59:51.639811 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:51.639695 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6e66819-f708-4e22-81d4-46adeef611e0-original-pull-secret\") pod \"global-pull-secret-syncer-54p5l\" (UID: \"a6e66819-f708-4e22-81d4-46adeef611e0\") " pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:59:51.642823 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:51.642805 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 02:59:51.642888 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:51.642873 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 02:59:51.652557 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:51.652531 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6e66819-f708-4e22-81d4-46adeef611e0-original-pull-secret\") pod \"global-pull-secret-syncer-54p5l\" (UID: \"a6e66819-f708-4e22-81d4-46adeef611e0\") " pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:59:51.652677 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:51.652653 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 02:59:51.663360 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:51.663332 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh64r\" (UniqueName: \"kubernetes.io/projected/36d7c073-6c29-456f-a540-34e61812ac39-kube-api-access-nh64r\") pod \"network-check-target-hldxf\" (UID: \"36d7c073-6c29-456f-a540-34e61812ac39\") " pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:59:51.690464 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:51.690428 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-54p5l" Jul 08 02:59:51.698227 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:51.698200 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fsx5r\"" Jul 08 02:59:51.705681 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:51.705648 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lsmbz" Jul 08 02:59:51.801156 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:51.801124 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v7mdx\"" Jul 08 02:59:51.807574 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:51.807436 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:59:51.838959 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:51.838907 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-54p5l"] Jul 08 02:59:51.842424 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:59:51.842394 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e66819_f708_4e22_81d4_46adeef611e0.slice/crio-6d85d065fb28d0a31db2deee7009bf03d83e6571bb42664edc0066f10ca7f0f5 WatchSource:0}: Error finding container 6d85d065fb28d0a31db2deee7009bf03d83e6571bb42664edc0066f10ca7f0f5: Status 404 returned error can't find the container with id 6d85d065fb28d0a31db2deee7009bf03d83e6571bb42664edc0066f10ca7f0f5 Jul 08 02:59:51.859884 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:51.859855 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lsmbz"] Jul 08 02:59:51.862935 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:59:51.862889 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5283b1_208f_415c_819f_c3e479102863.slice/crio-629895200ba85be19b3257a4650d5f454b2799c4126e833e0383a0938f06a9fe WatchSource:0}: Error finding container 629895200ba85be19b3257a4650d5f454b2799c4126e833e0383a0938f06a9fe: Status 404 returned error can't find the container with id 629895200ba85be19b3257a4650d5f454b2799c4126e833e0383a0938f06a9fe Jul 08 02:59:51.978091 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:51.978032 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hldxf"] Jul 08 02:59:51.982023 ip-10-0-128-70 kubenswrapper[2564]: W0708 02:59:51.981992 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36d7c073_6c29_456f_a540_34e61812ac39.slice/crio-a221e1b1dd5c5aa5a26b29a764bfcb626a07ccfd8336e2dd85c4244c9e9e54c3 WatchSource:0}: Error finding container a221e1b1dd5c5aa5a26b29a764bfcb626a07ccfd8336e2dd85c4244c9e9e54c3: Status 404 returned error can't find the container with id a221e1b1dd5c5aa5a26b29a764bfcb626a07ccfd8336e2dd85c4244c9e9e54c3 Jul 08 02:59:52.198485 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:52.198437 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lsmbz" event={"ID":"8f5283b1-208f-415c-819f-c3e479102863","Type":"ContainerStarted","Data":"629895200ba85be19b3257a4650d5f454b2799c4126e833e0383a0938f06a9fe"} Jul 08 02:59:52.199831 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:52.199802 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-54p5l" event={"ID":"a6e66819-f708-4e22-81d4-46adeef611e0","Type":"ContainerStarted","Data":"6d85d065fb28d0a31db2deee7009bf03d83e6571bb42664edc0066f10ca7f0f5"} Jul 08 02:59:52.201083 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:52.201058 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hldxf" event={"ID":"36d7c073-6c29-456f-a540-34e61812ac39","Type":"ContainerStarted","Data":"a221e1b1dd5c5aa5a26b29a764bfcb626a07ccfd8336e2dd85c4244c9e9e54c3"} Jul 08 02:59:53.121966 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:53.121908 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:53.222210 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:53.222174 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:54.220784 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:54.220745 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5141e7b4-0262-4a1d-809f-752a82e5f41f","Type":"ContainerStarted","Data":"7b28096353acba6c9988c7654db3f08e5b8da2308fb76674fba4eb7a421537d1"} Jul 08 02:59:54.221292 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:54.220850 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5141e7b4-0262-4a1d-809f-752a82e5f41f","Type":"ContainerStarted","Data":"a8ef0b97d02278d02aa7d3ac4ccab9a86fbd48abfaf5d4d4fef0d6e6a8cf24f3"} Jul 08 02:59:54.221292 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:54.220894 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5141e7b4-0262-4a1d-809f-752a82e5f41f","Type":"ContainerStarted","Data":"823574fd33ebf6094af52352bb667f1ddd555c43cc0dbd60d3bd2f87f829f9c8"} Jul 08 02:59:54.221435 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:54.221408 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5141e7b4-0262-4a1d-809f-752a82e5f41f","Type":"ContainerStarted","Data":"8b8d368c88cf04d2c0e1ff0bf0bcfca81da40275fd284aba54c71c705fea8bf6"} Jul 08 02:59:54.221494 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:54.221471 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5141e7b4-0262-4a1d-809f-752a82e5f41f","Type":"ContainerStarted","Data":"45398f374eb05ba3861a42cebcd9a77e72005655d24ad99cdf228a1d31547077"} Jul 08 02:59:54.221542 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:54.221511 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5141e7b4-0262-4a1d-809f-752a82e5f41f","Type":"ContainerStarted","Data":"3932f00c2c83d0a7282375561b0f9e21f2fe67ee6f6d25f96149dcf80f0a8630"} Jul 08 02:59:54.251776 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:54.251714 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.1721540360000002 podStartE2EDuration="6.251688813s" totalImagesPullingTime="3.061900392s" totalInitContainerRuntime="17.634385ms" isStatefulPod=false podCreationTimestamp="2026-07-08 02:59:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:54.249783016 +0000 UTC m=+68.907158928" watchObservedRunningTime="2026-07-08 02:59:54.251688813 +0000 UTC m=+68.909064686" Jul 08 02:59:54.304020 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:54.303982 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 02:59:54.542663 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:54.542576 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:54.594383 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:54.594298 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:54.664396 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:54.664350 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:54.720220 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:54.720159 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:55.233176 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:55.233139 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lsmbz" event={"ID":"8f5283b1-208f-415c-819f-c3e479102863","Type":"ContainerStarted","Data":"2b21a3b76deb3a673cbeaa7a643a8564ad130d8ebc827c7fa4423de2366d46ff"} Jul 08 02:59:55.233759 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:55.233727 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lsmbz" event={"ID":"8f5283b1-208f-415c-819f-c3e479102863","Type":"ContainerStarted","Data":"79598b418a27df7ba5b44dc90991962a98617e828b56203960c1b465653e5477"} Jul 08 02:59:55.250005 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:55.249954 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lsmbz" podStartSLOduration=67.31959761 podStartE2EDuration="1m10.249931782s" totalImagesPullingTime="2.930334172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:55.249168666 +0000 UTC m=+69.906544539" watchObservedRunningTime="2026-07-08 02:59:55.249931782 +0000 UTC m=+69.907307646" Jul 08 02:59:55.336107 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:55.335973 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:55.394336 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:55.394300 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:56.200041 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:56.200009 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-575dbdcf76-mt7x6" Jul 08 02:59:56.571884 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:56.571798 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:56.626605 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:56.626561 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:58.248749 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:58.248655 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-54p5l" event={"ID":"a6e66819-f708-4e22-81d4-46adeef611e0","Type":"ContainerStarted","Data":"dca4d3c3f83e36825ed002415eb4fb0a75ed97a4b86ce5f2077e3d14aba69353"} Jul 08 02:59:58.250321 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:58.250292 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hldxf" event={"ID":"36d7c073-6c29-456f-a540-34e61812ac39","Type":"ContainerStarted","Data":"445e932a832118cb7b335aa9fe08896ba5b8f8e5f2466c4d357ca8a2604260be"} Jul 08 02:59:58.250458 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:58.250352 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 02:59:58.264747 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:58.264700 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-54p5l" podStartSLOduration=67.166381727 podStartE2EDuration="1m13.264686558s" totalImagesPullingTime="6.098304831s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 02:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:58.264148845 +0000 UTC m=+72.921524745" watchObservedRunningTime="2026-07-08 02:59:58.264686558 +0000 UTC m=+72.922062428" Jul 08 02:59:58.281212 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:58.281155 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hldxf" podStartSLOduration=67.3252911 podStartE2EDuration="1m13.281132922s" totalImagesPullingTime="5.955841822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 02:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 02:59:58.280340755 +0000 UTC m=+72.937716626" watchObservedRunningTime="2026-07-08 02:59:58.281132922 +0000 UTC m=+72.938508841" Jul 08 02:59:58.727629 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:58.727598 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:58.829035 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:58.829005 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 02:59:58.929806 ip-10-0-128-70 kubenswrapper[2564]: I0708 02:59:58.929773 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:00:04.959451 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:00:04.959414 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:00:05.018872 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:00:05.018827 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:00:05.588351 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:00:05.588319 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:00:13.463279 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:00:13.463145 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:00:13.568765 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:00:13.568726 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:00:13.663732 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:00:13.663693 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:00:29.256742 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:00:29.256711 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hldxf" Jul 08 03:00:49.304479 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:00:49.304440 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:00:49.324840 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:00:49.324800 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:00:49.461416 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:00:49.461387 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:01.945285 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:01.945219 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:01:02.143205 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:02.143167 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:01:03.004364 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:03.004326 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 03:01:03.004787 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:03.004630 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerName="alertmanager" containerID="cri-o://db79e07a64fd78f2dea8cfbbe1355a31318404665dd5ce17bc41afe5c2e295f1" gracePeriod=120 Jul 08 03:01:03.004787 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:03.004704 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerName="kube-rbac-proxy-metric" containerID="cri-o://8a302955e9a7306c2a963a73d03dd19e0562c53f282166fcc80e2a4fc746242f" gracePeriod=120 Jul 08 03:01:03.004787 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:03.004746 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerName="kube-rbac-proxy-web" containerID="cri-o://0798b0ea7d27cfdeb45da811020926dff3c668bdfa26bcdbdcdd43721645695a" gracePeriod=120 Jul 08 03:01:03.004984 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:03.004791 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerName="kube-rbac-proxy" containerID="cri-o://3b992808db3fc67b03da8df1b5c23ce48ca2a2981795934c64cc8c62bfbbb550" gracePeriod=120 Jul 08 03:01:03.004984 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:03.004752 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerName="config-reloader" containerID="cri-o://1e70432784c57e12889d55e1823c8df41b29d634e355e94fce506f1935dec2e9" gracePeriod=120 Jul 08 03:01:03.004984 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:03.004960 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerName="prom-label-proxy" containerID="cri-o://75630a037d2f93a7d44ca42f357ce1c71ab9127cfe7c4b813e1e7a0c7d8b2279" gracePeriod=120 Jul 08 03:01:03.498388 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:03.498353 2564 generic.go:356] "Generic (PLEG): container finished" podID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerID="75630a037d2f93a7d44ca42f357ce1c71ab9127cfe7c4b813e1e7a0c7d8b2279" exitCode=0 Jul 08 03:01:03.498388 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:03.498385 2564 generic.go:356] "Generic (PLEG): container finished" podID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerID="8a302955e9a7306c2a963a73d03dd19e0562c53f282166fcc80e2a4fc746242f" exitCode=0 Jul 08 03:01:03.498388 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:03.498394 2564 generic.go:356] "Generic (PLEG): container finished" podID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerID="3b992808db3fc67b03da8df1b5c23ce48ca2a2981795934c64cc8c62bfbbb550" exitCode=0 Jul 08 03:01:03.498388 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:03.498401 2564 generic.go:356] "Generic (PLEG): container finished" podID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerID="1e70432784c57e12889d55e1823c8df41b29d634e355e94fce506f1935dec2e9" exitCode=0 Jul 08 03:01:03.498658 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:03.498409 2564 generic.go:356] "Generic (PLEG): container finished" podID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerID="db79e07a64fd78f2dea8cfbbe1355a31318404665dd5ce17bc41afe5c2e295f1" exitCode=0 Jul 08 03:01:03.498658 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:03.498430 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de0f3a7e-a05c-4274-bc1e-657236a9bfea","Type":"ContainerDied","Data":"75630a037d2f93a7d44ca42f357ce1c71ab9127cfe7c4b813e1e7a0c7d8b2279"} Jul 08 03:01:03.498658 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:03.498501 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de0f3a7e-a05c-4274-bc1e-657236a9bfea","Type":"ContainerDied","Data":"8a302955e9a7306c2a963a73d03dd19e0562c53f282166fcc80e2a4fc746242f"} Jul 08 03:01:03.498658 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:03.498529 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de0f3a7e-a05c-4274-bc1e-657236a9bfea","Type":"ContainerDied","Data":"3b992808db3fc67b03da8df1b5c23ce48ca2a2981795934c64cc8c62bfbbb550"} Jul 08 03:01:03.498658 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:03.498554 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de0f3a7e-a05c-4274-bc1e-657236a9bfea","Type":"ContainerDied","Data":"1e70432784c57e12889d55e1823c8df41b29d634e355e94fce506f1935dec2e9"} Jul 08 03:01:03.498658 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:03.498578 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de0f3a7e-a05c-4274-bc1e-657236a9bfea","Type":"ContainerDied","Data":"db79e07a64fd78f2dea8cfbbe1355a31318404665dd5ce17bc41afe5c2e295f1"} Jul 08 03:01:04.247734 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.247711 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.354698 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.354616 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-cluster-tls-config\") pod \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " Jul 08 03:01:04.354698 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.354660 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bltlt\" (UniqueName: \"kubernetes.io/projected/de0f3a7e-a05c-4274-bc1e-657236a9bfea-kube-api-access-bltlt\") pod \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " Jul 08 03:01:04.354698 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.354697 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/de0f3a7e-a05c-4274-bc1e-657236a9bfea-alertmanager-main-db\") pod \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " Jul 08 03:01:04.355004 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.354716 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de0f3a7e-a05c-4274-bc1e-657236a9bfea-alertmanager-trusted-ca-bundle\") pod \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " Jul 08 03:01:04.355004 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.354733 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-secret-alertmanager-kube-rbac-proxy-web\") pod \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " Jul 08 03:01:04.355004 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.354770 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-config-volume\") pod \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " Jul 08 03:01:04.355004 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.354791 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-web-config\") pod \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " Jul 08 03:01:04.355004 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.354821 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de0f3a7e-a05c-4274-bc1e-657236a9bfea-metrics-client-ca\") pod \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " Jul 08 03:01:04.355004 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.354865 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-secret-alertmanager-kube-rbac-proxy-metric\") pod \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " Jul 08 03:01:04.355004 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.354889 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de0f3a7e-a05c-4274-bc1e-657236a9bfea-tls-assets\") pod \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " Jul 08 03:01:04.355004 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.354933 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de0f3a7e-a05c-4274-bc1e-657236a9bfea-config-out\") pod \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " Jul 08 03:01:04.355004 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.354963 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-secret-alertmanager-main-tls\") pod \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " Jul 08 03:01:04.355446 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.355017 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-secret-alertmanager-kube-rbac-proxy\") pod \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\" (UID: \"de0f3a7e-a05c-4274-bc1e-657236a9bfea\") " Jul 08 03:01:04.355446 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.355102 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0f3a7e-a05c-4274-bc1e-657236a9bfea-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "de0f3a7e-a05c-4274-bc1e-657236a9bfea" (UID: "de0f3a7e-a05c-4274-bc1e-657236a9bfea"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:01:04.355446 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.355209 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de0f3a7e-a05c-4274-bc1e-657236a9bfea-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "de0f3a7e-a05c-4274-bc1e-657236a9bfea" (UID: "de0f3a7e-a05c-4274-bc1e-657236a9bfea"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:01:04.355603 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.355511 2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/de0f3a7e-a05c-4274-bc1e-657236a9bfea-alertmanager-main-db\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:04.355603 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.355530 2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de0f3a7e-a05c-4274-bc1e-657236a9bfea-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:04.356037 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.355983 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de0f3a7e-a05c-4274-bc1e-657236a9bfea-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "de0f3a7e-a05c-4274-bc1e-657236a9bfea" (UID: "de0f3a7e-a05c-4274-bc1e-657236a9bfea"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:01:04.358331 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.358256 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "de0f3a7e-a05c-4274-bc1e-657236a9bfea" (UID: "de0f3a7e-a05c-4274-bc1e-657236a9bfea"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:01:04.358616 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.358523 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "de0f3a7e-a05c-4274-bc1e-657236a9bfea" (UID: "de0f3a7e-a05c-4274-bc1e-657236a9bfea"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:01:04.358797 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.358765 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "de0f3a7e-a05c-4274-bc1e-657236a9bfea" (UID: "de0f3a7e-a05c-4274-bc1e-657236a9bfea"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:01:04.359155 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.359078 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-config-volume" (OuterVolumeSpecName: "config-volume") pod "de0f3a7e-a05c-4274-bc1e-657236a9bfea" (UID: "de0f3a7e-a05c-4274-bc1e-657236a9bfea"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:01:04.359155 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.359132 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0f3a7e-a05c-4274-bc1e-657236a9bfea-config-out" (OuterVolumeSpecName: "config-out") pod "de0f3a7e-a05c-4274-bc1e-657236a9bfea" (UID: "de0f3a7e-a05c-4274-bc1e-657236a9bfea"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:01:04.359155 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.359137 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de0f3a7e-a05c-4274-bc1e-657236a9bfea-kube-api-access-bltlt" (OuterVolumeSpecName: "kube-api-access-bltlt") pod "de0f3a7e-a05c-4274-bc1e-657236a9bfea" (UID: "de0f3a7e-a05c-4274-bc1e-657236a9bfea"). InnerVolumeSpecName "kube-api-access-bltlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:01:04.359780 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.359750 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de0f3a7e-a05c-4274-bc1e-657236a9bfea-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "de0f3a7e-a05c-4274-bc1e-657236a9bfea" (UID: "de0f3a7e-a05c-4274-bc1e-657236a9bfea"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:01:04.360273 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.360254 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "de0f3a7e-a05c-4274-bc1e-657236a9bfea" (UID: "de0f3a7e-a05c-4274-bc1e-657236a9bfea"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:01:04.362415 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.362371 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "de0f3a7e-a05c-4274-bc1e-657236a9bfea" (UID: "de0f3a7e-a05c-4274-bc1e-657236a9bfea"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:01:04.369375 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.369265 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-web-config" (OuterVolumeSpecName: "web-config") pod "de0f3a7e-a05c-4274-bc1e-657236a9bfea" (UID: "de0f3a7e-a05c-4274-bc1e-657236a9bfea"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:01:04.456370 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.456331 2564 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-cluster-tls-config\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:04.456370 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.456361 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bltlt\" (UniqueName: \"kubernetes.io/projected/de0f3a7e-a05c-4274-bc1e-657236a9bfea-kube-api-access-bltlt\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:04.456370 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.456373 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:04.456370 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.456383 2564 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-config-volume\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:04.456880 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.456392 2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-web-config\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:04.456880 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.456399 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de0f3a7e-a05c-4274-bc1e-657236a9bfea-metrics-client-ca\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:04.456880 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.456407 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:04.456880 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.456416 2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de0f3a7e-a05c-4274-bc1e-657236a9bfea-tls-assets\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:04.456880 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.456424 2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de0f3a7e-a05c-4274-bc1e-657236a9bfea-config-out\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:04.456880 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.456432 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-secret-alertmanager-main-tls\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:04.456880 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.456440 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de0f3a7e-a05c-4274-bc1e-657236a9bfea-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:04.504948 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.504887 2564 generic.go:356] "Generic (PLEG): container finished" podID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerID="0798b0ea7d27cfdeb45da811020926dff3c668bdfa26bcdbdcdd43721645695a" exitCode=0 Jul 08 03:01:04.505112 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.504957 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de0f3a7e-a05c-4274-bc1e-657236a9bfea","Type":"ContainerDied","Data":"0798b0ea7d27cfdeb45da811020926dff3c668bdfa26bcdbdcdd43721645695a"} Jul 08 03:01:04.505112 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.504984 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.505112 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.505030 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de0f3a7e-a05c-4274-bc1e-657236a9bfea","Type":"ContainerDied","Data":"51ef688c1d0b2d110f7dfb2c3efbcbf524f0ba2113a0f1cc57c50f2662c1d10c"} Jul 08 03:01:04.505112 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.505063 2564 scope.go:117] "RemoveContainer" containerID="75630a037d2f93a7d44ca42f357ce1c71ab9127cfe7c4b813e1e7a0c7d8b2279" Jul 08 03:01:04.513691 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.513674 2564 scope.go:117] "RemoveContainer" containerID="8a302955e9a7306c2a963a73d03dd19e0562c53f282166fcc80e2a4fc746242f" Jul 08 03:01:04.520657 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.520638 2564 scope.go:117] "RemoveContainer" containerID="3b992808db3fc67b03da8df1b5c23ce48ca2a2981795934c64cc8c62bfbbb550" Jul 08 03:01:04.527511 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.527483 2564 scope.go:117] "RemoveContainer" containerID="0798b0ea7d27cfdeb45da811020926dff3c668bdfa26bcdbdcdd43721645695a" Jul 08 03:01:04.532654 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.532631 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 03:01:04.535314 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.535293 2564 scope.go:117] "RemoveContainer" containerID="1e70432784c57e12889d55e1823c8df41b29d634e355e94fce506f1935dec2e9" Jul 08 03:01:04.542255 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.542232 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 03:01:04.542852 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.542831 2564 scope.go:117] "RemoveContainer" containerID="db79e07a64fd78f2dea8cfbbe1355a31318404665dd5ce17bc41afe5c2e295f1" Jul 08 03:01:04.550060 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.550040 2564 scope.go:117] "RemoveContainer" containerID="8c29c8f036afd01a5fbd4388075ac0a1380da38a04f6e2898559b9b32cd2c280" Jul 08 03:01:04.557321 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.557302 2564 scope.go:117] "RemoveContainer" containerID="75630a037d2f93a7d44ca42f357ce1c71ab9127cfe7c4b813e1e7a0c7d8b2279" Jul 08 03:01:04.557661 ip-10-0-128-70 kubenswrapper[2564]: E0708 03:01:04.557635 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75630a037d2f93a7d44ca42f357ce1c71ab9127cfe7c4b813e1e7a0c7d8b2279\": container with ID starting with 75630a037d2f93a7d44ca42f357ce1c71ab9127cfe7c4b813e1e7a0c7d8b2279 not found: ID does not exist" containerID="75630a037d2f93a7d44ca42f357ce1c71ab9127cfe7c4b813e1e7a0c7d8b2279" Jul 08 03:01:04.557842 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.557669 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75630a037d2f93a7d44ca42f357ce1c71ab9127cfe7c4b813e1e7a0c7d8b2279"} err="failed to get container status \"75630a037d2f93a7d44ca42f357ce1c71ab9127cfe7c4b813e1e7a0c7d8b2279\": rpc error: code = NotFound desc = could not find container \"75630a037d2f93a7d44ca42f357ce1c71ab9127cfe7c4b813e1e7a0c7d8b2279\": container with ID starting with 75630a037d2f93a7d44ca42f357ce1c71ab9127cfe7c4b813e1e7a0c7d8b2279 not found: ID does not exist" Jul 08 03:01:04.557842 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.557705 2564 scope.go:117] "RemoveContainer" containerID="8a302955e9a7306c2a963a73d03dd19e0562c53f282166fcc80e2a4fc746242f" Jul 08 03:01:04.557983 ip-10-0-128-70 kubenswrapper[2564]: E0708 03:01:04.557963 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a302955e9a7306c2a963a73d03dd19e0562c53f282166fcc80e2a4fc746242f\": container with ID starting with 8a302955e9a7306c2a963a73d03dd19e0562c53f282166fcc80e2a4fc746242f not found: ID does not exist" containerID="8a302955e9a7306c2a963a73d03dd19e0562c53f282166fcc80e2a4fc746242f" Jul 08 03:01:04.558042 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.557995 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a302955e9a7306c2a963a73d03dd19e0562c53f282166fcc80e2a4fc746242f"} err="failed to get container status \"8a302955e9a7306c2a963a73d03dd19e0562c53f282166fcc80e2a4fc746242f\": rpc error: code = NotFound desc = could not find container \"8a302955e9a7306c2a963a73d03dd19e0562c53f282166fcc80e2a4fc746242f\": container with ID starting with 8a302955e9a7306c2a963a73d03dd19e0562c53f282166fcc80e2a4fc746242f not found: ID does not exist" Jul 08 03:01:04.558042 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.558022 2564 scope.go:117] "RemoveContainer" containerID="3b992808db3fc67b03da8df1b5c23ce48ca2a2981795934c64cc8c62bfbbb550" Jul 08 03:01:04.558264 ip-10-0-128-70 kubenswrapper[2564]: E0708 03:01:04.558249 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b992808db3fc67b03da8df1b5c23ce48ca2a2981795934c64cc8c62bfbbb550\": container with ID starting with 3b992808db3fc67b03da8df1b5c23ce48ca2a2981795934c64cc8c62bfbbb550 not found: ID does not exist" containerID="3b992808db3fc67b03da8df1b5c23ce48ca2a2981795934c64cc8c62bfbbb550" Jul 08 03:01:04.558303 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.558271 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b992808db3fc67b03da8df1b5c23ce48ca2a2981795934c64cc8c62bfbbb550"} err="failed to get container status \"3b992808db3fc67b03da8df1b5c23ce48ca2a2981795934c64cc8c62bfbbb550\": rpc error: code = NotFound desc = could not find container \"3b992808db3fc67b03da8df1b5c23ce48ca2a2981795934c64cc8c62bfbbb550\": container with ID starting with 3b992808db3fc67b03da8df1b5c23ce48ca2a2981795934c64cc8c62bfbbb550 not found: ID does not exist" Jul 08 03:01:04.558303 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.558291 2564 scope.go:117] "RemoveContainer" containerID="0798b0ea7d27cfdeb45da811020926dff3c668bdfa26bcdbdcdd43721645695a" Jul 08 03:01:04.558510 ip-10-0-128-70 kubenswrapper[2564]: E0708 03:01:04.558494 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0798b0ea7d27cfdeb45da811020926dff3c668bdfa26bcdbdcdd43721645695a\": container with ID starting with 0798b0ea7d27cfdeb45da811020926dff3c668bdfa26bcdbdcdd43721645695a not found: ID does not exist" containerID="0798b0ea7d27cfdeb45da811020926dff3c668bdfa26bcdbdcdd43721645695a" Jul 08 03:01:04.558552 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.558515 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0798b0ea7d27cfdeb45da811020926dff3c668bdfa26bcdbdcdd43721645695a"} err="failed to get container status \"0798b0ea7d27cfdeb45da811020926dff3c668bdfa26bcdbdcdd43721645695a\": rpc error: code = NotFound desc = could not find container \"0798b0ea7d27cfdeb45da811020926dff3c668bdfa26bcdbdcdd43721645695a\": container with ID starting with 0798b0ea7d27cfdeb45da811020926dff3c668bdfa26bcdbdcdd43721645695a not found: ID does not exist" Jul 08 03:01:04.558552 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.558531 2564 scope.go:117] "RemoveContainer" containerID="1e70432784c57e12889d55e1823c8df41b29d634e355e94fce506f1935dec2e9" Jul 08 03:01:04.558753 ip-10-0-128-70 kubenswrapper[2564]: E0708 03:01:04.558735 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e70432784c57e12889d55e1823c8df41b29d634e355e94fce506f1935dec2e9\": container with ID starting with 1e70432784c57e12889d55e1823c8df41b29d634e355e94fce506f1935dec2e9 not found: ID does not exist" containerID="1e70432784c57e12889d55e1823c8df41b29d634e355e94fce506f1935dec2e9" Jul 08 03:01:04.558821 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.558761 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e70432784c57e12889d55e1823c8df41b29d634e355e94fce506f1935dec2e9"} err="failed to get container status \"1e70432784c57e12889d55e1823c8df41b29d634e355e94fce506f1935dec2e9\": rpc error: code = NotFound desc = could not find container \"1e70432784c57e12889d55e1823c8df41b29d634e355e94fce506f1935dec2e9\": container with ID starting with 1e70432784c57e12889d55e1823c8df41b29d634e355e94fce506f1935dec2e9 not found: ID does not exist" Jul 08 03:01:04.558821 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.558781 2564 scope.go:117] "RemoveContainer" containerID="db79e07a64fd78f2dea8cfbbe1355a31318404665dd5ce17bc41afe5c2e295f1" Jul 08 03:01:04.559019 ip-10-0-128-70 kubenswrapper[2564]: E0708 03:01:04.558997 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db79e07a64fd78f2dea8cfbbe1355a31318404665dd5ce17bc41afe5c2e295f1\": container with ID starting with db79e07a64fd78f2dea8cfbbe1355a31318404665dd5ce17bc41afe5c2e295f1 not found: ID does not exist" containerID="db79e07a64fd78f2dea8cfbbe1355a31318404665dd5ce17bc41afe5c2e295f1" Jul 08 03:01:04.559064 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.559023 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db79e07a64fd78f2dea8cfbbe1355a31318404665dd5ce17bc41afe5c2e295f1"} err="failed to get container status \"db79e07a64fd78f2dea8cfbbe1355a31318404665dd5ce17bc41afe5c2e295f1\": rpc error: code = NotFound desc = could not find container \"db79e07a64fd78f2dea8cfbbe1355a31318404665dd5ce17bc41afe5c2e295f1\": container with ID starting with db79e07a64fd78f2dea8cfbbe1355a31318404665dd5ce17bc41afe5c2e295f1 not found: ID does not exist" Jul 08 03:01:04.559064 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.559035 2564 scope.go:117] "RemoveContainer" containerID="8c29c8f036afd01a5fbd4388075ac0a1380da38a04f6e2898559b9b32cd2c280" Jul 08 03:01:04.559244 ip-10-0-128-70 kubenswrapper[2564]: E0708 03:01:04.559228 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c29c8f036afd01a5fbd4388075ac0a1380da38a04f6e2898559b9b32cd2c280\": container with ID starting with 8c29c8f036afd01a5fbd4388075ac0a1380da38a04f6e2898559b9b32cd2c280 not found: ID does not exist" containerID="8c29c8f036afd01a5fbd4388075ac0a1380da38a04f6e2898559b9b32cd2c280" Jul 08 03:01:04.559288 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.559248 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c29c8f036afd01a5fbd4388075ac0a1380da38a04f6e2898559b9b32cd2c280"} err="failed to get container status \"8c29c8f036afd01a5fbd4388075ac0a1380da38a04f6e2898559b9b32cd2c280\": rpc error: code = NotFound desc = could not find container \"8c29c8f036afd01a5fbd4388075ac0a1380da38a04f6e2898559b9b32cd2c280\": container with ID starting with 8c29c8f036afd01a5fbd4388075ac0a1380da38a04f6e2898559b9b32cd2c280 not found: ID does not exist" Jul 08 03:01:04.570792 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.570764 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 03:01:04.571064 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.571051 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerName="kube-rbac-proxy-web" Jul 08 03:01:04.571117 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.571067 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerName="kube-rbac-proxy-web" Jul 08 03:01:04.571117 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.571075 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerName="prom-label-proxy" Jul 08 03:01:04.571117 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.571082 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerName="prom-label-proxy" Jul 08 03:01:04.571117 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.571089 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerName="config-reloader" Jul 08 03:01:04.571117 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.571095 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerName="config-reloader" Jul 08 03:01:04.571117 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.571105 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerName="init-config-reloader" Jul 08 03:01:04.571117 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.571111 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerName="init-config-reloader" Jul 08 03:01:04.571117 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.571117 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerName="kube-rbac-proxy" Jul 08 03:01:04.571341 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.571122 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerName="kube-rbac-proxy" Jul 08 03:01:04.571341 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.571129 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerName="kube-rbac-proxy-metric" Jul 08 03:01:04.571341 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.571135 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerName="kube-rbac-proxy-metric" Jul 08 03:01:04.571341 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.571148 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerName="alertmanager" Jul 08 03:01:04.571341 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.571153 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerName="alertmanager" Jul 08 03:01:04.571341 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.571197 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerName="alertmanager" Jul 08 03:01:04.571341 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.571206 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerName="kube-rbac-proxy-metric" Jul 08 03:01:04.571341 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.571212 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerName="kube-rbac-proxy-web" Jul 08 03:01:04.571341 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.571216 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerName="prom-label-proxy" Jul 08 03:01:04.571341 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.571222 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerName="kube-rbac-proxy" Jul 08 03:01:04.571341 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.571228 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" containerName="config-reloader" Jul 08 03:01:04.574935 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.574901 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.579063 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.578064 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 08 03:01:04.579063 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.578329 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 08 03:01:04.579063 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.578553 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 08 03:01:04.579063 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.578811 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xdpxj\"" Jul 08 03:01:04.579063 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.579007 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 08 03:01:04.579507 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.579345 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 08 03:01:04.579507 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.579405 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 08 03:01:04.579664 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.579556 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 08 03:01:04.579664 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.579607 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 08 03:01:04.583300 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.583275 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 08 03:01:04.589623 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.589601 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 03:01:04.658283 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.658243 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/878d90d4-98a5-4075-a5ec-3f20b859838a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.658283 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.658288 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/878d90d4-98a5-4075-a5ec-3f20b859838a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.658549 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.658306 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/878d90d4-98a5-4075-a5ec-3f20b859838a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.658549 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.658324 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/878d90d4-98a5-4075-a5ec-3f20b859838a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.658549 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.658390 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/878d90d4-98a5-4075-a5ec-3f20b859838a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.658549 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.658432 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/878d90d4-98a5-4075-a5ec-3f20b859838a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.658549 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.658463 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/878d90d4-98a5-4075-a5ec-3f20b859838a-config-volume\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.658549 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.658496 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/878d90d4-98a5-4075-a5ec-3f20b859838a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.658549 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.658513 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p4wc\" (UniqueName: \"kubernetes.io/projected/878d90d4-98a5-4075-a5ec-3f20b859838a-kube-api-access-6p4wc\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.658549 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.658529 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/878d90d4-98a5-4075-a5ec-3f20b859838a-config-out\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.658549 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.658545 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/878d90d4-98a5-4075-a5ec-3f20b859838a-web-config\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.658802 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.658608 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/878d90d4-98a5-4075-a5ec-3f20b859838a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.658802 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.658631 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/878d90d4-98a5-4075-a5ec-3f20b859838a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.759668 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.759634 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/878d90d4-98a5-4075-a5ec-3f20b859838a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.759761 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.759677 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/878d90d4-98a5-4075-a5ec-3f20b859838a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.759761 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.759695 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/878d90d4-98a5-4075-a5ec-3f20b859838a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.759761 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.759726 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/878d90d4-98a5-4075-a5ec-3f20b859838a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.759873 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.759758 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/878d90d4-98a5-4075-a5ec-3f20b859838a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.759873 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.759787 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/878d90d4-98a5-4075-a5ec-3f20b859838a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.759873 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.759822 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/878d90d4-98a5-4075-a5ec-3f20b859838a-config-volume\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.759873 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.759852 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/878d90d4-98a5-4075-a5ec-3f20b859838a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.760097 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.759874 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p4wc\" (UniqueName: \"kubernetes.io/projected/878d90d4-98a5-4075-a5ec-3f20b859838a-kube-api-access-6p4wc\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.760097 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.759895 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/878d90d4-98a5-4075-a5ec-3f20b859838a-config-out\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.760097 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.759913 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/878d90d4-98a5-4075-a5ec-3f20b859838a-web-config\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.760097 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.759993 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/878d90d4-98a5-4075-a5ec-3f20b859838a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.760097 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.760019 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/878d90d4-98a5-4075-a5ec-3f20b859838a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.760349 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.760324 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/878d90d4-98a5-4075-a5ec-3f20b859838a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.761615 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.761115 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/878d90d4-98a5-4075-a5ec-3f20b859838a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.761615 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.761273 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/878d90d4-98a5-4075-a5ec-3f20b859838a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.762860 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.762807 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/878d90d4-98a5-4075-a5ec-3f20b859838a-config-out\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.762981 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.762943 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/878d90d4-98a5-4075-a5ec-3f20b859838a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.763100 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.763081 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/878d90d4-98a5-4075-a5ec-3f20b859838a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.763289 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.763268 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/878d90d4-98a5-4075-a5ec-3f20b859838a-config-volume\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.763447 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.763427 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/878d90d4-98a5-4075-a5ec-3f20b859838a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.763624 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.763608 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/878d90d4-98a5-4075-a5ec-3f20b859838a-web-config\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.763662 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.763623 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/878d90d4-98a5-4075-a5ec-3f20b859838a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.764282 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.764264 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/878d90d4-98a5-4075-a5ec-3f20b859838a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.764760 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.764745 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/878d90d4-98a5-4075-a5ec-3f20b859838a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.772449 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.772427 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p4wc\" (UniqueName: \"kubernetes.io/projected/878d90d4-98a5-4075-a5ec-3f20b859838a-kube-api-access-6p4wc\") pod \"alertmanager-main-0\" (UID: \"878d90d4-98a5-4075-a5ec-3f20b859838a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:04.886245 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:04.886207 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 03:01:05.051424 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:05.051385 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 03:01:05.056332 ip-10-0-128-70 kubenswrapper[2564]: W0708 03:01:05.056300 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod878d90d4_98a5_4075_a5ec_3f20b859838a.slice/crio-a486f64327474f22bdb5d93274db10780b67614912bb63b1b53b1282406a4c28 WatchSource:0}: Error finding container a486f64327474f22bdb5d93274db10780b67614912bb63b1b53b1282406a4c28: Status 404 returned error can't find the container with id a486f64327474f22bdb5d93274db10780b67614912bb63b1b53b1282406a4c28 Jul 08 03:01:05.512978 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:05.512938 2564 generic.go:356] "Generic (PLEG): container finished" podID="878d90d4-98a5-4075-a5ec-3f20b859838a" containerID="05f17fe7b5793c90d2d9a7109271e07f58a62a818f334cd1bfe8eee23da6f24e" exitCode=0 Jul 08 03:01:05.513343 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:05.513030 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"878d90d4-98a5-4075-a5ec-3f20b859838a","Type":"ContainerDied","Data":"05f17fe7b5793c90d2d9a7109271e07f58a62a818f334cd1bfe8eee23da6f24e"} Jul 08 03:01:05.513343 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:05.513113 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"878d90d4-98a5-4075-a5ec-3f20b859838a","Type":"ContainerStarted","Data":"a486f64327474f22bdb5d93274db10780b67614912bb63b1b53b1282406a4c28"} Jul 08 03:01:05.886643 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:05.886610 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de0f3a7e-a05c-4274-bc1e-657236a9bfea" path="/var/lib/kubelet/pods/de0f3a7e-a05c-4274-bc1e-657236a9bfea/volumes" Jul 08 03:01:06.520104 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:06.520073 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"878d90d4-98a5-4075-a5ec-3f20b859838a","Type":"ContainerStarted","Data":"f25756aa5ea57480c173f33a41b215433844442fb67c8b6324894f980315e59f"} Jul 08 03:01:06.520557 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:06.520136 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"878d90d4-98a5-4075-a5ec-3f20b859838a","Type":"ContainerStarted","Data":"208c69bbf90986159ee2c8ec5d36c8853730ff2ee1d9ae85004326f839de2ab5"} Jul 08 03:01:06.520557 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:06.520164 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"878d90d4-98a5-4075-a5ec-3f20b859838a","Type":"ContainerStarted","Data":"33841405a72d7cda6a8bb120a09f94d492486eea87908d0eb7815122598a6218"} Jul 08 03:01:06.520557 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:06.520188 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"878d90d4-98a5-4075-a5ec-3f20b859838a","Type":"ContainerStarted","Data":"cb54fdffb80a36543280fe3216aa4ebb0dea6a38df1d8b91c63cf1d3f218192a"} Jul 08 03:01:06.520557 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:06.520213 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"878d90d4-98a5-4075-a5ec-3f20b859838a","Type":"ContainerStarted","Data":"001f9b9e38708ae779f4123bf50fa260f4f72e2780ed9e18d6bcc981d047ef60"} Jul 08 03:01:06.520557 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:06.520235 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"878d90d4-98a5-4075-a5ec-3f20b859838a","Type":"ContainerStarted","Data":"3a96381f5369fd425a0292e8c42b9f538128c0e4316dd3b92d166caf21bd1be3"} Jul 08 03:01:06.552714 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:06.552625 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.538724498 podStartE2EDuration="2.552598293s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.873795ms" isStatefulPod=false podCreationTimestamp="2026-07-08 03:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:01:06.550358414 +0000 UTC m=+141.207734322" watchObservedRunningTime="2026-07-08 03:01:06.552598293 +0000 UTC m=+141.209974162" Jul 08 03:01:07.338803 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.338761 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 03:01:07.339243 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.339208 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerName="prometheus" containerID="cri-o://3932f00c2c83d0a7282375561b0f9e21f2fe67ee6f6d25f96149dcf80f0a8630" gracePeriod=600 Jul 08 03:01:07.339614 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.339587 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerName="kube-rbac-proxy-thanos" containerID="cri-o://7b28096353acba6c9988c7654db3f08e5b8da2308fb76674fba4eb7a421537d1" gracePeriod=600 Jul 08 03:01:07.339724 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.339704 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerName="kube-rbac-proxy" containerID="cri-o://a8ef0b97d02278d02aa7d3ac4ccab9a86fbd48abfaf5d4d4fef0d6e6a8cf24f3" gracePeriod=600 Jul 08 03:01:07.339808 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.339788 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerName="kube-rbac-proxy-web" containerID="cri-o://823574fd33ebf6094af52352bb667f1ddd555c43cc0dbd60d3bd2f87f829f9c8" gracePeriod=600 Jul 08 03:01:07.339886 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.339867 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerName="thanos-sidecar" containerID="cri-o://8b8d368c88cf04d2c0e1ff0bf0bcfca81da40275fd284aba54c71c705fea8bf6" gracePeriod=600 Jul 08 03:01:07.339985 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.339968 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerName="config-reloader" containerID="cri-o://45398f374eb05ba3861a42cebcd9a77e72005655d24ad99cdf228a1d31547077" gracePeriod=600 Jul 08 03:01:07.530112 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.530077 2564 generic.go:356] "Generic (PLEG): container finished" podID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerID="7b28096353acba6c9988c7654db3f08e5b8da2308fb76674fba4eb7a421537d1" exitCode=0 Jul 08 03:01:07.530112 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.530107 2564 generic.go:356] "Generic (PLEG): container finished" podID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerID="a8ef0b97d02278d02aa7d3ac4ccab9a86fbd48abfaf5d4d4fef0d6e6a8cf24f3" exitCode=0 Jul 08 03:01:07.530112 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.530116 2564 generic.go:356] "Generic (PLEG): container finished" podID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerID="823574fd33ebf6094af52352bb667f1ddd555c43cc0dbd60d3bd2f87f829f9c8" exitCode=0 Jul 08 03:01:07.530669 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.530127 2564 generic.go:356] "Generic (PLEG): container finished" podID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerID="8b8d368c88cf04d2c0e1ff0bf0bcfca81da40275fd284aba54c71c705fea8bf6" exitCode=0 Jul 08 03:01:07.530669 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.530137 2564 generic.go:356] "Generic (PLEG): container finished" podID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerID="45398f374eb05ba3861a42cebcd9a77e72005655d24ad99cdf228a1d31547077" exitCode=0 Jul 08 03:01:07.530669 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.530146 2564 generic.go:356] "Generic (PLEG): container finished" podID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerID="3932f00c2c83d0a7282375561b0f9e21f2fe67ee6f6d25f96149dcf80f0a8630" exitCode=0 Jul 08 03:01:07.532902 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.530721 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5141e7b4-0262-4a1d-809f-752a82e5f41f","Type":"ContainerDied","Data":"7b28096353acba6c9988c7654db3f08e5b8da2308fb76674fba4eb7a421537d1"} Jul 08 03:01:07.533063 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.532984 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5141e7b4-0262-4a1d-809f-752a82e5f41f","Type":"ContainerDied","Data":"a8ef0b97d02278d02aa7d3ac4ccab9a86fbd48abfaf5d4d4fef0d6e6a8cf24f3"} Jul 08 03:01:07.533063 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.533035 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5141e7b4-0262-4a1d-809f-752a82e5f41f","Type":"ContainerDied","Data":"823574fd33ebf6094af52352bb667f1ddd555c43cc0dbd60d3bd2f87f829f9c8"} Jul 08 03:01:07.533173 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.533074 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5141e7b4-0262-4a1d-809f-752a82e5f41f","Type":"ContainerDied","Data":"8b8d368c88cf04d2c0e1ff0bf0bcfca81da40275fd284aba54c71c705fea8bf6"} Jul 08 03:01:07.533173 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.533112 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5141e7b4-0262-4a1d-809f-752a82e5f41f","Type":"ContainerDied","Data":"45398f374eb05ba3861a42cebcd9a77e72005655d24ad99cdf228a1d31547077"} Jul 08 03:01:07.533173 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.533150 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5141e7b4-0262-4a1d-809f-752a82e5f41f","Type":"ContainerDied","Data":"3932f00c2c83d0a7282375561b0f9e21f2fe67ee6f6d25f96149dcf80f0a8630"} Jul 08 03:01:07.596047 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.595982 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:07.686807 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.686776 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-metrics-client-certs\") pod \"5141e7b4-0262-4a1d-809f-752a82e5f41f\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " Jul 08 03:01:07.687029 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.686836 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-configmap-serving-certs-ca-bundle\") pod \"5141e7b4-0262-4a1d-809f-752a82e5f41f\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " Jul 08 03:01:07.687029 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.686873 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sz6s\" (UniqueName: \"kubernetes.io/projected/5141e7b4-0262-4a1d-809f-752a82e5f41f-kube-api-access-2sz6s\") pod \"5141e7b4-0262-4a1d-809f-752a82e5f41f\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " Jul 08 03:01:07.687140 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.687050 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-configmap-metrics-client-ca\") pod \"5141e7b4-0262-4a1d-809f-752a82e5f41f\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " Jul 08 03:01:07.687140 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.687095 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-k8s-rulefiles-0\") pod \"5141e7b4-0262-4a1d-809f-752a82e5f41f\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " Jul 08 03:01:07.687140 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.687126 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-k8s-rulefiles-2\") pod \"5141e7b4-0262-4a1d-809f-752a82e5f41f\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " Jul 08 03:01:07.687311 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.687144 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-k8s-db\") pod \"5141e7b4-0262-4a1d-809f-752a82e5f41f\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " Jul 08 03:01:07.687311 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.687180 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-k8s-rulefiles-1\") pod \"5141e7b4-0262-4a1d-809f-752a82e5f41f\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " Jul 08 03:01:07.687311 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.687208 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-thanos-prometheus-http-client-file\") pod \"5141e7b4-0262-4a1d-809f-752a82e5f41f\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " Jul 08 03:01:07.687311 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.687240 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-kube-rbac-proxy\") pod \"5141e7b4-0262-4a1d-809f-752a82e5f41f\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " Jul 08 03:01:07.687311 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.687279 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-configmap-kubelet-serving-ca-bundle\") pod \"5141e7b4-0262-4a1d-809f-752a82e5f41f\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " Jul 08 03:01:07.687567 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.687312 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"5141e7b4-0262-4a1d-809f-752a82e5f41f\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " Jul 08 03:01:07.687567 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.687328 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "5141e7b4-0262-4a1d-809f-752a82e5f41f" (UID: "5141e7b4-0262-4a1d-809f-752a82e5f41f"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:01:07.687567 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.687338 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-prometheus-k8s-tls\") pod \"5141e7b4-0262-4a1d-809f-752a82e5f41f\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " Jul 08 03:01:07.687567 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.687419 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5141e7b4-0262-4a1d-809f-752a82e5f41f-tls-assets\") pod \"5141e7b4-0262-4a1d-809f-752a82e5f41f\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " Jul 08 03:01:07.687567 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.687448 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"5141e7b4-0262-4a1d-809f-752a82e5f41f\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " Jul 08 03:01:07.687567 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.687491 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-grpc-tls\") pod \"5141e7b4-0262-4a1d-809f-752a82e5f41f\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " Jul 08 03:01:07.687567 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.687521 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-config\") pod \"5141e7b4-0262-4a1d-809f-752a82e5f41f\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " Jul 08 03:01:07.687567 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.687561 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5141e7b4-0262-4a1d-809f-752a82e5f41f-config-out\") pod \"5141e7b4-0262-4a1d-809f-752a82e5f41f\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " Jul 08 03:01:07.687972 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.687599 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-web-config\") pod \"5141e7b4-0262-4a1d-809f-752a82e5f41f\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " Jul 08 03:01:07.687972 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.687628 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-trusted-ca-bundle\") pod \"5141e7b4-0262-4a1d-809f-752a82e5f41f\" (UID: \"5141e7b4-0262-4a1d-809f-752a82e5f41f\") " Jul 08 03:01:07.687972 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.687574 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "5141e7b4-0262-4a1d-809f-752a82e5f41f" (UID: "5141e7b4-0262-4a1d-809f-752a82e5f41f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:01:07.687972 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.687856 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "5141e7b4-0262-4a1d-809f-752a82e5f41f" (UID: "5141e7b4-0262-4a1d-809f-752a82e5f41f"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:01:07.688178 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.688138 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:07.688178 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.688158 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-configmap-metrics-client-ca\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:07.688178 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.688173 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:07.690065 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.688714 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "5141e7b4-0262-4a1d-809f-752a82e5f41f" (UID: "5141e7b4-0262-4a1d-809f-752a82e5f41f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:01:07.690065 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.689410 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "5141e7b4-0262-4a1d-809f-752a82e5f41f" (UID: "5141e7b4-0262-4a1d-809f-752a82e5f41f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:01:07.690065 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.689691 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "5141e7b4-0262-4a1d-809f-752a82e5f41f" (UID: "5141e7b4-0262-4a1d-809f-752a82e5f41f"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:01:07.690065 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.689988 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "5141e7b4-0262-4a1d-809f-752a82e5f41f" (UID: "5141e7b4-0262-4a1d-809f-752a82e5f41f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:01:07.690354 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.690135 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "5141e7b4-0262-4a1d-809f-752a82e5f41f" (UID: "5141e7b4-0262-4a1d-809f-752a82e5f41f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:01:07.690723 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.690695 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "5141e7b4-0262-4a1d-809f-752a82e5f41f" (UID: "5141e7b4-0262-4a1d-809f-752a82e5f41f"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:01:07.690949 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.690907 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "5141e7b4-0262-4a1d-809f-752a82e5f41f" (UID: "5141e7b4-0262-4a1d-809f-752a82e5f41f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:01:07.691566 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.691541 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5141e7b4-0262-4a1d-809f-752a82e5f41f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5141e7b4-0262-4a1d-809f-752a82e5f41f" (UID: "5141e7b4-0262-4a1d-809f-752a82e5f41f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:01:07.692422 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.692395 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "5141e7b4-0262-4a1d-809f-752a82e5f41f" (UID: "5141e7b4-0262-4a1d-809f-752a82e5f41f"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:01:07.692674 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.692614 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "5141e7b4-0262-4a1d-809f-752a82e5f41f" (UID: "5141e7b4-0262-4a1d-809f-752a82e5f41f"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:01:07.692867 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.692846 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5141e7b4-0262-4a1d-809f-752a82e5f41f-kube-api-access-2sz6s" (OuterVolumeSpecName: "kube-api-access-2sz6s") pod "5141e7b4-0262-4a1d-809f-752a82e5f41f" (UID: "5141e7b4-0262-4a1d-809f-752a82e5f41f"). InnerVolumeSpecName "kube-api-access-2sz6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:01:07.693094 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.693068 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "5141e7b4-0262-4a1d-809f-752a82e5f41f" (UID: "5141e7b4-0262-4a1d-809f-752a82e5f41f"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:01:07.693387 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.693348 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-config" (OuterVolumeSpecName: "config") pod "5141e7b4-0262-4a1d-809f-752a82e5f41f" (UID: "5141e7b4-0262-4a1d-809f-752a82e5f41f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:01:07.693768 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.693741 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "5141e7b4-0262-4a1d-809f-752a82e5f41f" (UID: "5141e7b4-0262-4a1d-809f-752a82e5f41f"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:01:07.694286 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.694257 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "5141e7b4-0262-4a1d-809f-752a82e5f41f" (UID: "5141e7b4-0262-4a1d-809f-752a82e5f41f"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:01:07.694743 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.694720 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5141e7b4-0262-4a1d-809f-752a82e5f41f-config-out" (OuterVolumeSpecName: "config-out") pod "5141e7b4-0262-4a1d-809f-752a82e5f41f" (UID: "5141e7b4-0262-4a1d-809f-752a82e5f41f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:01:07.703491 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.703455 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-web-config" (OuterVolumeSpecName: "web-config") pod "5141e7b4-0262-4a1d-809f-752a82e5f41f" (UID: "5141e7b4-0262-4a1d-809f-752a82e5f41f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:01:07.789504 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.789461 2564 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-grpc-tls\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:07.789504 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.789498 2564 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-config\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:07.789504 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.789506 2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5141e7b4-0262-4a1d-809f-752a82e5f41f-config-out\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:07.789504 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.789514 2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-web-config\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:07.789504 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.789523 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:07.789793 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.789533 2564 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-metrics-client-certs\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:07.789793 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.789541 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sz6s\" (UniqueName: \"kubernetes.io/projected/5141e7b4-0262-4a1d-809f-752a82e5f41f-kube-api-access-2sz6s\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:07.789793 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.789550 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:07.789793 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.789558 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-k8s-db\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:07.789793 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.789566 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:07.789793 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.789577 2564 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:07.789793 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.789584 2564 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-kube-rbac-proxy\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:07.789793 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.789594 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5141e7b4-0262-4a1d-809f-752a82e5f41f-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:07.789793 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.789603 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:07.789793 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.789612 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:07.789793 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.789621 2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5141e7b4-0262-4a1d-809f-752a82e5f41f-tls-assets\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:07.789793 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:07.789629 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5141e7b4-0262-4a1d-809f-752a82e5f41f-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:01:08.537915 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.537885 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5141e7b4-0262-4a1d-809f-752a82e5f41f","Type":"ContainerDied","Data":"43d9016c8a7a7a3440cf9c5dc639232076f946df89fe578f31c9a2c905e7887b"} Jul 08 03:01:08.538344 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.537963 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.538344 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.537974 2564 scope.go:117] "RemoveContainer" containerID="7b28096353acba6c9988c7654db3f08e5b8da2308fb76674fba4eb7a421537d1" Jul 08 03:01:08.545650 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.545627 2564 scope.go:117] "RemoveContainer" containerID="a8ef0b97d02278d02aa7d3ac4ccab9a86fbd48abfaf5d4d4fef0d6e6a8cf24f3" Jul 08 03:01:08.552820 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.552799 2564 scope.go:117] "RemoveContainer" containerID="823574fd33ebf6094af52352bb667f1ddd555c43cc0dbd60d3bd2f87f829f9c8" Jul 08 03:01:08.559721 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.559702 2564 scope.go:117] "RemoveContainer" containerID="8b8d368c88cf04d2c0e1ff0bf0bcfca81da40275fd284aba54c71c705fea8bf6" Jul 08 03:01:08.567324 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.567303 2564 scope.go:117] "RemoveContainer" containerID="45398f374eb05ba3861a42cebcd9a77e72005655d24ad99cdf228a1d31547077" Jul 08 03:01:08.567431 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.567411 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 03:01:08.573347 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.573327 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 03:01:08.574678 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.574656 2564 scope.go:117] "RemoveContainer" containerID="3932f00c2c83d0a7282375561b0f9e21f2fe67ee6f6d25f96149dcf80f0a8630" Jul 08 03:01:08.581834 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.581815 2564 scope.go:117] "RemoveContainer" containerID="731bb7c7f40b76b5b5a6e519787af4971c6fd088111ad0fb687a77dbe9cc6b81" Jul 08 03:01:08.612377 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.612271 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 03:01:08.612651 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.612633 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerName="init-config-reloader" Jul 08 03:01:08.612754 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.612655 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerName="init-config-reloader" Jul 08 03:01:08.612754 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.612667 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerName="prometheus" Jul 08 03:01:08.612754 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.612675 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerName="prometheus" Jul 08 03:01:08.612754 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.612687 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerName="kube-rbac-proxy-thanos" Jul 08 03:01:08.612754 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.612695 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerName="kube-rbac-proxy-thanos" Jul 08 03:01:08.612754 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.612710 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerName="thanos-sidecar" Jul 08 03:01:08.612754 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.612718 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerName="thanos-sidecar" Jul 08 03:01:08.612754 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.612727 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerName="kube-rbac-proxy-web" Jul 08 03:01:08.612754 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.612735 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerName="kube-rbac-proxy-web" Jul 08 03:01:08.612754 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.612746 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerName="kube-rbac-proxy" Jul 08 03:01:08.612754 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.612755 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerName="kube-rbac-proxy" Jul 08 03:01:08.613321 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.612774 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerName="config-reloader" Jul 08 03:01:08.613321 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.612783 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerName="config-reloader" Jul 08 03:01:08.613321 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.612861 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerName="kube-rbac-proxy-web" Jul 08 03:01:08.613321 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.612875 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerName="thanos-sidecar" Jul 08 03:01:08.613321 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.612886 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerName="prometheus" Jul 08 03:01:08.613321 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.612895 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerName="config-reloader" Jul 08 03:01:08.613321 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.612902 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerName="kube-rbac-proxy" Jul 08 03:01:08.613321 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.612909 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5141e7b4-0262-4a1d-809f-752a82e5f41f" containerName="kube-rbac-proxy-thanos" Jul 08 03:01:08.616853 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.616832 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.619774 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.619752 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 08 03:01:08.619891 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.619876 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tk22q\"" Jul 08 03:01:08.622237 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.622219 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 08 03:01:08.622351 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.622287 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bkaojbjkcl3af\"" Jul 08 03:01:08.622490 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.622472 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 08 03:01:08.622547 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.622538 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 08 03:01:08.623692 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.623667 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 08 03:01:08.623781 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.623753 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 08 03:01:08.624032 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.624014 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 08 03:01:08.624140 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.624121 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 08 03:01:08.624203 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.624140 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 08 03:01:08.624259 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.624092 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 08 03:01:08.624259 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.624100 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 08 03:01:08.624259 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.624053 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 08 03:01:08.625032 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.625008 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 08 03:01:08.626622 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.626602 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 08 03:01:08.634394 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.634347 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 08 03:01:08.639769 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.639743 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 03:01:08.696452 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.696411 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ba3abfa-0a04-476f-8354-0f02bd55bf08-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.696452 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.696452 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0ba3abfa-0a04-476f-8354-0f02bd55bf08-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.696702 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.696469 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ba3abfa-0a04-476f-8354-0f02bd55bf08-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.696702 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.696487 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ba3abfa-0a04-476f-8354-0f02bd55bf08-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.696702 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.696507 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0ba3abfa-0a04-476f-8354-0f02bd55bf08-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.696702 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.696536 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0ba3abfa-0a04-476f-8354-0f02bd55bf08-web-config\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.696702 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.696566 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7lqw\" (UniqueName: \"kubernetes.io/projected/0ba3abfa-0a04-476f-8354-0f02bd55bf08-kube-api-access-n7lqw\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.696702 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.696583 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0ba3abfa-0a04-476f-8354-0f02bd55bf08-config\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.696702 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.696598 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0ba3abfa-0a04-476f-8354-0f02bd55bf08-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.696702 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.696615 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ba3abfa-0a04-476f-8354-0f02bd55bf08-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.696702 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.696632 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0ba3abfa-0a04-476f-8354-0f02bd55bf08-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.696702 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.696659 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0ba3abfa-0a04-476f-8354-0f02bd55bf08-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.696702 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.696676 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0ba3abfa-0a04-476f-8354-0f02bd55bf08-config-out\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.696702 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.696689 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0ba3abfa-0a04-476f-8354-0f02bd55bf08-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.697099 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.696708 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0ba3abfa-0a04-476f-8354-0f02bd55bf08-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.697099 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.696726 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0ba3abfa-0a04-476f-8354-0f02bd55bf08-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.697099 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.696743 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ba3abfa-0a04-476f-8354-0f02bd55bf08-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.697099 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.696796 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0ba3abfa-0a04-476f-8354-0f02bd55bf08-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.697099 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.696847 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ba3abfa-0a04-476f-8354-0f02bd55bf08-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.697099 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.696873 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0ba3abfa-0a04-476f-8354-0f02bd55bf08-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.798232 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.798201 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0ba3abfa-0a04-476f-8354-0f02bd55bf08-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.798232 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.798240 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ba3abfa-0a04-476f-8354-0f02bd55bf08-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.798496 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.798256 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0ba3abfa-0a04-476f-8354-0f02bd55bf08-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.798496 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.798276 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ba3abfa-0a04-476f-8354-0f02bd55bf08-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.798496 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.798392 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ba3abfa-0a04-476f-8354-0f02bd55bf08-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.798496 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.798420 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0ba3abfa-0a04-476f-8354-0f02bd55bf08-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.798496 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.798440 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0ba3abfa-0a04-476f-8354-0f02bd55bf08-web-config\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.798496 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.798476 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7lqw\" (UniqueName: \"kubernetes.io/projected/0ba3abfa-0a04-476f-8354-0f02bd55bf08-kube-api-access-n7lqw\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.798782 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.798503 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0ba3abfa-0a04-476f-8354-0f02bd55bf08-config\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.798782 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.798523 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0ba3abfa-0a04-476f-8354-0f02bd55bf08-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.798782 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.798549 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ba3abfa-0a04-476f-8354-0f02bd55bf08-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.798782 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.798577 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0ba3abfa-0a04-476f-8354-0f02bd55bf08-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.798782 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.798626 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0ba3abfa-0a04-476f-8354-0f02bd55bf08-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.798782 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.798654 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0ba3abfa-0a04-476f-8354-0f02bd55bf08-config-out\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.798782 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.798677 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0ba3abfa-0a04-476f-8354-0f02bd55bf08-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.798782 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.798706 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0ba3abfa-0a04-476f-8354-0f02bd55bf08-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.798782 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.798736 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0ba3abfa-0a04-476f-8354-0f02bd55bf08-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.798782 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.798763 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ba3abfa-0a04-476f-8354-0f02bd55bf08-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.799287 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.798798 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0ba3abfa-0a04-476f-8354-0f02bd55bf08-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.799287 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.798835 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ba3abfa-0a04-476f-8354-0f02bd55bf08-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.799287 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.798995 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0ba3abfa-0a04-476f-8354-0f02bd55bf08-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.799287 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.799026 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0ba3abfa-0a04-476f-8354-0f02bd55bf08-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.799287 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.799057 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ba3abfa-0a04-476f-8354-0f02bd55bf08-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.800742 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.799543 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ba3abfa-0a04-476f-8354-0f02bd55bf08-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.800742 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.799615 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0ba3abfa-0a04-476f-8354-0f02bd55bf08-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.800742 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.800314 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ba3abfa-0a04-476f-8354-0f02bd55bf08-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.800742 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.800694 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ba3abfa-0a04-476f-8354-0f02bd55bf08-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.802984 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.802957 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0ba3abfa-0a04-476f-8354-0f02bd55bf08-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.803133 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.803113 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ba3abfa-0a04-476f-8354-0f02bd55bf08-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.803204 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.803153 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0ba3abfa-0a04-476f-8354-0f02bd55bf08-web-config\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.803819 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.803480 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0ba3abfa-0a04-476f-8354-0f02bd55bf08-config-out\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.803819 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.803662 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0ba3abfa-0a04-476f-8354-0f02bd55bf08-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.804001 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.803904 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0ba3abfa-0a04-476f-8354-0f02bd55bf08-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.804571 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.804549 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ba3abfa-0a04-476f-8354-0f02bd55bf08-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.804652 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.804568 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0ba3abfa-0a04-476f-8354-0f02bd55bf08-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.804723 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.804701 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0ba3abfa-0a04-476f-8354-0f02bd55bf08-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.804776 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.804738 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0ba3abfa-0a04-476f-8354-0f02bd55bf08-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.804991 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.804973 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0ba3abfa-0a04-476f-8354-0f02bd55bf08-config\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.805258 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.805242 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0ba3abfa-0a04-476f-8354-0f02bd55bf08-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.810432 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.810409 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7lqw\" (UniqueName: \"kubernetes.io/projected/0ba3abfa-0a04-476f-8354-0f02bd55bf08-kube-api-access-n7lqw\") pod \"prometheus-k8s-0\" (UID: \"0ba3abfa-0a04-476f-8354-0f02bd55bf08\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:08.926679 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:08.926645 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:09.066153 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:09.066118 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 03:01:09.068850 ip-10-0-128-70 kubenswrapper[2564]: W0708 03:01:09.068819 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ba3abfa_0a04_476f_8354_0f02bd55bf08.slice/crio-a9100b6275bb5e90254ec814afaaaf025d35b9963bcb41e5a8d61cab8e11fa1e WatchSource:0}: Error finding container a9100b6275bb5e90254ec814afaaaf025d35b9963bcb41e5a8d61cab8e11fa1e: Status 404 returned error can't find the container with id a9100b6275bb5e90254ec814afaaaf025d35b9963bcb41e5a8d61cab8e11fa1e Jul 08 03:01:09.543095 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:09.543058 2564 generic.go:356] "Generic (PLEG): container finished" podID="0ba3abfa-0a04-476f-8354-0f02bd55bf08" containerID="510764efc37c896ab4fff33f0cb9decbd44829b7ff2dea20def819de48d0c963" exitCode=0 Jul 08 03:01:09.543567 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:09.543156 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0ba3abfa-0a04-476f-8354-0f02bd55bf08","Type":"ContainerDied","Data":"510764efc37c896ab4fff33f0cb9decbd44829b7ff2dea20def819de48d0c963"} Jul 08 03:01:09.543567 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:09.543239 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0ba3abfa-0a04-476f-8354-0f02bd55bf08","Type":"ContainerStarted","Data":"a9100b6275bb5e90254ec814afaaaf025d35b9963bcb41e5a8d61cab8e11fa1e"} Jul 08 03:01:09.886292 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:09.886270 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5141e7b4-0262-4a1d-809f-752a82e5f41f" path="/var/lib/kubelet/pods/5141e7b4-0262-4a1d-809f-752a82e5f41f/volumes" Jul 08 03:01:10.553199 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:10.553160 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0ba3abfa-0a04-476f-8354-0f02bd55bf08","Type":"ContainerStarted","Data":"fad63141205f598de8d9233a53a0aac3e97e36868a4beace375b879720fdd644"} Jul 08 03:01:10.553663 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:10.553239 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0ba3abfa-0a04-476f-8354-0f02bd55bf08","Type":"ContainerStarted","Data":"51b6055d6f4c60e8d9b403083a212632b6a7c66e29d1c6dd27f4289b1795e059"} Jul 08 03:01:10.553663 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:10.553287 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0ba3abfa-0a04-476f-8354-0f02bd55bf08","Type":"ContainerStarted","Data":"e9dab7b506ee4cef9ae1e8e88863324b23c8d0e58be0ddd71670e5a32f38e2d0"} Jul 08 03:01:10.553663 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:10.553327 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0ba3abfa-0a04-476f-8354-0f02bd55bf08","Type":"ContainerStarted","Data":"d6e70d3483b03812b74316fd0626e03cc32959017db40a34d7069a88b9b5d6f7"} Jul 08 03:01:10.553663 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:10.553374 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0ba3abfa-0a04-476f-8354-0f02bd55bf08","Type":"ContainerStarted","Data":"b1f4377f834b912e6e4a0499aa8513e31d70e5322b10cf2e3557ec193ca2984e"} Jul 08 03:01:10.553663 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:10.553412 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0ba3abfa-0a04-476f-8354-0f02bd55bf08","Type":"ContainerStarted","Data":"998a0336868a3e083c082d22bcaec77fc88d20b2f987ce39d01377186a00e8ff"} Jul 08 03:01:10.590609 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:10.590543 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.5688005350000003 podStartE2EDuration="2.590525704s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.725169ms" isStatefulPod=false podCreationTimestamp="2026-07-08 03:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:01:10.588586986 +0000 UTC m=+145.245962858" watchObservedRunningTime="2026-07-08 03:01:10.590525704 +0000 UTC m=+145.247901576" Jul 08 03:01:12.895006 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:12.894970 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:01:12.951776 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:12.951744 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:01:13.927810 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:13.927761 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:01:24.001292 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:24.001256 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:01:24.072625 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:24.072585 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:01:24.135109 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:24.135080 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:01:37.735679 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:37.735608 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:01:58.437350 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:58.437314 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:01:58.491531 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:01:58.491501 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:02:08.927826 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:02:08.927780 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:02:08.944010 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:02:08.943913 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:02:09.785684 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:02:09.785659 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 03:02:17.195479 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:02:17.195449 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:02:17.265210 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:02:17.265167 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:02:27.821117 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:02:27.821077 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:02:27.874503 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:02:27.874468 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:02:27.959155 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:02:27.959119 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:02:28.012938 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:02:28.012881 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:02:28.560401 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:02:28.560367 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:02:28.617484 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:02:28.617453 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:02:29.136312 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:02:29.136254 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:02:29.237242 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:02:29.237201 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:02:34.966538 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:02:34.966499 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:02:35.067180 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:02:35.067125 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:02:35.167368 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:02:35.167338 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:02:38.279606 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:02:38.279572 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:02:38.336635 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:02:38.336606 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:02:38.396025 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:02:38.395994 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:02:41.996620 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:02:41.996585 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:03:12.130406 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:03:12.130300 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:03:12.219151 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:03:12.219119 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:03:45.727045 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:03:45.727019 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 08 03:04:57.522051 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:04:57.521642 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:04:59.408200 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:04:59.408163 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:01.052780 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:01.052743 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:01.186967 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:01.186914 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:02.611910 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:02.611877 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:02.709767 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:02.709727 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:03.105033 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:03.104998 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:03.225792 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:03.225762 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:04.983528 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:04.983496 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:05.398495 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:05.398461 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:08.243456 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:08.243418 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:08.308963 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:08.308932 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:10.799133 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:10.799092 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:10.898182 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:10.898147 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:12.203304 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:12.203270 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:16.443654 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:16.443622 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:16.541217 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:16.541184 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:17.863203 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:17.863168 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mwcwh"] Jul 08 03:05:17.866362 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:17.866345 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mwcwh" Jul 08 03:05:17.869116 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:17.869097 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 08 03:05:17.870168 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:17.870149 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 08 03:05:17.870276 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:17.870179 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-rrcsz\"" Jul 08 03:05:17.873898 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:17.873877 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mwcwh"] Jul 08 03:05:17.934834 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:17.934799 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9bf4a7d2-fa7c-4447-b950-b03527ca9d3e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mwcwh\" (UID: \"9bf4a7d2-fa7c-4447-b950-b03527ca9d3e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mwcwh" Jul 08 03:05:17.935048 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:17.934864 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p6cj\" (UniqueName: \"kubernetes.io/projected/9bf4a7d2-fa7c-4447-b950-b03527ca9d3e-kube-api-access-2p6cj\") pod \"openshift-lws-operator-fd8ccff4c-mwcwh\" (UID: \"9bf4a7d2-fa7c-4447-b950-b03527ca9d3e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mwcwh" Jul 08 03:05:18.036199 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:18.036147 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9bf4a7d2-fa7c-4447-b950-b03527ca9d3e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mwcwh\" (UID: \"9bf4a7d2-fa7c-4447-b950-b03527ca9d3e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mwcwh" Jul 08 03:05:18.036398 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:18.036219 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p6cj\" (UniqueName: \"kubernetes.io/projected/9bf4a7d2-fa7c-4447-b950-b03527ca9d3e-kube-api-access-2p6cj\") pod \"openshift-lws-operator-fd8ccff4c-mwcwh\" (UID: \"9bf4a7d2-fa7c-4447-b950-b03527ca9d3e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mwcwh" Jul 08 03:05:18.036612 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:18.036590 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9bf4a7d2-fa7c-4447-b950-b03527ca9d3e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mwcwh\" (UID: \"9bf4a7d2-fa7c-4447-b950-b03527ca9d3e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mwcwh" Jul 08 03:05:18.045624 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:18.045591 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p6cj\" (UniqueName: \"kubernetes.io/projected/9bf4a7d2-fa7c-4447-b950-b03527ca9d3e-kube-api-access-2p6cj\") pod \"openshift-lws-operator-fd8ccff4c-mwcwh\" (UID: \"9bf4a7d2-fa7c-4447-b950-b03527ca9d3e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mwcwh" Jul 08 03:05:18.186139 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:18.186099 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mwcwh" Jul 08 03:05:18.319381 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:18.319351 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mwcwh"] Jul 08 03:05:18.322231 ip-10-0-128-70 kubenswrapper[2564]: W0708 03:05:18.322197 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bf4a7d2_fa7c_4447_b950_b03527ca9d3e.slice/crio-3262ff4126a4df3b1327c9f0ba1ae27cf1f785837ae82130ceee2aff04c9a9ca WatchSource:0}: Error finding container 3262ff4126a4df3b1327c9f0ba1ae27cf1f785837ae82130ceee2aff04c9a9ca: Status 404 returned error can't find the container with id 3262ff4126a4df3b1327c9f0ba1ae27cf1f785837ae82130ceee2aff04c9a9ca Jul 08 03:05:18.323860 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:18.323843 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 03:05:18.450675 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:18.450593 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mwcwh" event={"ID":"9bf4a7d2-fa7c-4447-b950-b03527ca9d3e","Type":"ContainerStarted","Data":"3262ff4126a4df3b1327c9f0ba1ae27cf1f785837ae82130ceee2aff04c9a9ca"} Jul 08 03:05:21.470871 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:21.470826 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mwcwh" event={"ID":"9bf4a7d2-fa7c-4447-b950-b03527ca9d3e","Type":"ContainerStarted","Data":"54515116cfc20257adc133c0ef899e5c590074a0713687f69cbbcabf4f6f884e"} Jul 08 03:05:21.488557 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:21.488501 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mwcwh" podStartSLOduration=1.753691092 podStartE2EDuration="4.488486408s" totalImagesPullingTime="2.734795316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:05:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:05:21.486969443 +0000 UTC m=+396.144345314" watchObservedRunningTime="2026-07-08 03:05:21.488486408 +0000 UTC m=+396.145862279" Jul 08 03:05:22.173175 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:22.173135 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:22.274810 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:22.274771 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:24.223500 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:24.223470 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:29.751269 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:29.751233 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:37.651376 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:37.651326 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:38.730653 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:38.730567 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:38.828701 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:38.828671 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-cfk9b"] Jul 08 03:05:38.829200 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:38.829180 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:38.836196 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:38.836170 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-cfk9b" Jul 08 03:05:38.839220 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:38.839194 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 08 03:05:38.839220 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:38.839212 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-47wqr\"" Jul 08 03:05:38.839403 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:38.839212 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 08 03:05:38.841861 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:38.841827 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 08 03:05:38.842000 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:38.841863 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 08 03:05:38.861978 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:38.861952 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-cfk9b"] Jul 08 03:05:38.920243 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:38.920208 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4d827dbf-0236-4f9c-b93b-5ed89ada804f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-cfk9b\" (UID: \"4d827dbf-0236-4f9c-b93b-5ed89ada804f\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-cfk9b" Jul 08 03:05:38.920243 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:38.920259 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d827dbf-0236-4f9c-b93b-5ed89ada804f-webhook-cert\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-cfk9b\" (UID: \"4d827dbf-0236-4f9c-b93b-5ed89ada804f\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-cfk9b" Jul 08 03:05:38.920455 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:38.920280 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czl8c\" (UniqueName: \"kubernetes.io/projected/4d827dbf-0236-4f9c-b93b-5ed89ada804f-kube-api-access-czl8c\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-cfk9b\" (UID: \"4d827dbf-0236-4f9c-b93b-5ed89ada804f\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-cfk9b" Jul 08 03:05:38.920455 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:38.920360 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4d827dbf-0236-4f9c-b93b-5ed89ada804f-manifests\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-cfk9b\" (UID: \"4d827dbf-0236-4f9c-b93b-5ed89ada804f\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-cfk9b" Jul 08 03:05:38.920455 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:38.920444 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d827dbf-0236-4f9c-b93b-5ed89ada804f-tmp\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-cfk9b\" (UID: \"4d827dbf-0236-4f9c-b93b-5ed89ada804f\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-cfk9b" Jul 08 03:05:38.997723 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:38.997641 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:39.021752 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:39.021721 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4d827dbf-0236-4f9c-b93b-5ed89ada804f-manifests\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-cfk9b\" (UID: \"4d827dbf-0236-4f9c-b93b-5ed89ada804f\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-cfk9b" Jul 08 03:05:39.021910 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:39.021776 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d827dbf-0236-4f9c-b93b-5ed89ada804f-tmp\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-cfk9b\" (UID: \"4d827dbf-0236-4f9c-b93b-5ed89ada804f\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-cfk9b" Jul 08 03:05:39.021910 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:39.021808 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4d827dbf-0236-4f9c-b93b-5ed89ada804f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-cfk9b\" (UID: \"4d827dbf-0236-4f9c-b93b-5ed89ada804f\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-cfk9b" Jul 08 03:05:39.021910 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:39.021829 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d827dbf-0236-4f9c-b93b-5ed89ada804f-webhook-cert\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-cfk9b\" (UID: \"4d827dbf-0236-4f9c-b93b-5ed89ada804f\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-cfk9b" Jul 08 03:05:39.021910 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:39.021847 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czl8c\" (UniqueName: \"kubernetes.io/projected/4d827dbf-0236-4f9c-b93b-5ed89ada804f-kube-api-access-czl8c\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-cfk9b\" (UID: \"4d827dbf-0236-4f9c-b93b-5ed89ada804f\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-cfk9b" Jul 08 03:05:39.022191 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:39.022159 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4d827dbf-0236-4f9c-b93b-5ed89ada804f-manifests\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-cfk9b\" (UID: \"4d827dbf-0236-4f9c-b93b-5ed89ada804f\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-cfk9b" Jul 08 03:05:39.024419 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:39.024397 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d827dbf-0236-4f9c-b93b-5ed89ada804f-tmp\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-cfk9b\" (UID: \"4d827dbf-0236-4f9c-b93b-5ed89ada804f\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-cfk9b" Jul 08 03:05:39.024528 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:39.024507 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4d827dbf-0236-4f9c-b93b-5ed89ada804f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-cfk9b\" (UID: \"4d827dbf-0236-4f9c-b93b-5ed89ada804f\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-cfk9b" Jul 08 03:05:39.024639 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:39.024618 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d827dbf-0236-4f9c-b93b-5ed89ada804f-webhook-cert\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-cfk9b\" (UID: \"4d827dbf-0236-4f9c-b93b-5ed89ada804f\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-cfk9b" Jul 08 03:05:39.030970 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:39.030948 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czl8c\" (UniqueName: \"kubernetes.io/projected/4d827dbf-0236-4f9c-b93b-5ed89ada804f-kube-api-access-czl8c\") pod \"opendatahub-operator-controller-manager-64b8ffddd6-cfk9b\" (UID: \"4d827dbf-0236-4f9c-b93b-5ed89ada804f\") " pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-cfk9b" Jul 08 03:05:39.098141 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:39.098101 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:39.147384 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:39.147352 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-cfk9b" Jul 08 03:05:39.202136 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:39.202105 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:39.276688 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:39.276604 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:39.286432 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:39.286412 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-cfk9b"] Jul 08 03:05:39.289126 ip-10-0-128-70 kubenswrapper[2564]: W0708 03:05:39.289101 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d827dbf_0236_4f9c_b93b_5ed89ada804f.slice/crio-8b5bf1a9949d4441bfb116c59584d49ce85008b97013b39ef730c49f18c0c29d WatchSource:0}: Error finding container 8b5bf1a9949d4441bfb116c59584d49ce85008b97013b39ef730c49f18c0c29d: Status 404 returned error can't find the container with id 8b5bf1a9949d4441bfb116c59584d49ce85008b97013b39ef730c49f18c0c29d Jul 08 03:05:39.339816 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:39.339778 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:39.437141 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:39.437107 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:39.541193 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:39.541097 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-cfk9b" event={"ID":"4d827dbf-0236-4f9c-b93b-5ed89ada804f","Type":"ContainerStarted","Data":"8b5bf1a9949d4441bfb116c59584d49ce85008b97013b39ef730c49f18c0c29d"} Jul 08 03:05:42.555382 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:42.555291 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-cfk9b" event={"ID":"4d827dbf-0236-4f9c-b93b-5ed89ada804f","Type":"ContainerStarted","Data":"35e9b423627e4bf334a17066f2696289d1834d4e7f6a74e79cf346294eda554c"} Jul 08 03:05:43.560533 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:43.560498 2564 generic.go:356] "Generic (PLEG): container finished" podID="4d827dbf-0236-4f9c-b93b-5ed89ada804f" containerID="35e9b423627e4bf334a17066f2696289d1834d4e7f6a74e79cf346294eda554c" exitCode=0 Jul 08 03:05:43.560898 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:43.560592 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-cfk9b" event={"ID":"4d827dbf-0236-4f9c-b93b-5ed89ada804f","Type":"ContainerDied","Data":"35e9b423627e4bf334a17066f2696289d1834d4e7f6a74e79cf346294eda554c"} Jul 08 03:05:44.566479 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:44.566447 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-cfk9b" event={"ID":"4d827dbf-0236-4f9c-b93b-5ed89ada804f","Type":"ContainerStarted","Data":"bb0b1b4950d01e8d043c90423d5efa4dbf88d8dadf553929b17f46b6bd86ea43"} Jul 08 03:05:44.566871 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:44.566568 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-cfk9b" Jul 08 03:05:44.591278 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:44.591220 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-cfk9b" podStartSLOduration=2.01422698 podStartE2EDuration="6.591204614s" totalImagesPullingTime="3.548387332s" totalInitContainerRuntime="1.028590302s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:05:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:05:44.588764248 +0000 UTC m=+419.246140110" watchObservedRunningTime="2026-07-08 03:05:44.591204614 +0000 UTC m=+419.248580533" Jul 08 03:05:50.513569 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:50.513525 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:50.606558 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:50.606503 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:50.868158 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:50.868077 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:50.968941 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:50.968884 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:55.572853 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:55.572826 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-64b8ffddd6-cfk9b" Jul 08 03:05:56.812587 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:56.812556 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-65cfbc7dbc-cwswd"] Jul 08 03:05:56.824578 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:56.824548 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-65cfbc7dbc-cwswd" Jul 08 03:05:56.835972 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:56.835897 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 08 03:05:56.836212 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:56.836185 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 08 03:05:56.836447 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:56.836426 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-d46gp\"" Jul 08 03:05:56.836673 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:56.836649 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 08 03:05:56.838170 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:56.838148 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-65cfbc7dbc-cwswd"] Jul 08 03:05:56.977036 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:56.976981 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b965c3f1-ed80-4b1f-8448-3df5096345e0-cert\") pod \"lws-controller-manager-65cfbc7dbc-cwswd\" (UID: \"b965c3f1-ed80-4b1f-8448-3df5096345e0\") " pod="openshift-lws-operator/lws-controller-manager-65cfbc7dbc-cwswd" Jul 08 03:05:56.977036 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:56.977045 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b965c3f1-ed80-4b1f-8448-3df5096345e0-metrics-cert\") pod \"lws-controller-manager-65cfbc7dbc-cwswd\" (UID: \"b965c3f1-ed80-4b1f-8448-3df5096345e0\") " pod="openshift-lws-operator/lws-controller-manager-65cfbc7dbc-cwswd" Jul 08 03:05:56.977306 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:56.977065 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b965c3f1-ed80-4b1f-8448-3df5096345e0-manager-config\") pod \"lws-controller-manager-65cfbc7dbc-cwswd\" (UID: \"b965c3f1-ed80-4b1f-8448-3df5096345e0\") " pod="openshift-lws-operator/lws-controller-manager-65cfbc7dbc-cwswd" Jul 08 03:05:56.977306 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:56.977158 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9pn9\" (UniqueName: \"kubernetes.io/projected/b965c3f1-ed80-4b1f-8448-3df5096345e0-kube-api-access-t9pn9\") pod \"lws-controller-manager-65cfbc7dbc-cwswd\" (UID: \"b965c3f1-ed80-4b1f-8448-3df5096345e0\") " pod="openshift-lws-operator/lws-controller-manager-65cfbc7dbc-cwswd" Jul 08 03:05:57.078608 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:57.078520 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b965c3f1-ed80-4b1f-8448-3df5096345e0-cert\") pod \"lws-controller-manager-65cfbc7dbc-cwswd\" (UID: \"b965c3f1-ed80-4b1f-8448-3df5096345e0\") " pod="openshift-lws-operator/lws-controller-manager-65cfbc7dbc-cwswd" Jul 08 03:05:57.078608 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:57.078571 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b965c3f1-ed80-4b1f-8448-3df5096345e0-metrics-cert\") pod \"lws-controller-manager-65cfbc7dbc-cwswd\" (UID: \"b965c3f1-ed80-4b1f-8448-3df5096345e0\") " pod="openshift-lws-operator/lws-controller-manager-65cfbc7dbc-cwswd" Jul 08 03:05:57.078608 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:57.078588 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b965c3f1-ed80-4b1f-8448-3df5096345e0-manager-config\") pod \"lws-controller-manager-65cfbc7dbc-cwswd\" (UID: \"b965c3f1-ed80-4b1f-8448-3df5096345e0\") " pod="openshift-lws-operator/lws-controller-manager-65cfbc7dbc-cwswd" Jul 08 03:05:57.078875 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:57.078622 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9pn9\" (UniqueName: \"kubernetes.io/projected/b965c3f1-ed80-4b1f-8448-3df5096345e0-kube-api-access-t9pn9\") pod \"lws-controller-manager-65cfbc7dbc-cwswd\" (UID: \"b965c3f1-ed80-4b1f-8448-3df5096345e0\") " pod="openshift-lws-operator/lws-controller-manager-65cfbc7dbc-cwswd" Jul 08 03:05:57.079300 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:57.079276 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b965c3f1-ed80-4b1f-8448-3df5096345e0-manager-config\") pod \"lws-controller-manager-65cfbc7dbc-cwswd\" (UID: \"b965c3f1-ed80-4b1f-8448-3df5096345e0\") " pod="openshift-lws-operator/lws-controller-manager-65cfbc7dbc-cwswd" Jul 08 03:05:57.081282 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:57.081262 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b965c3f1-ed80-4b1f-8448-3df5096345e0-metrics-cert\") pod \"lws-controller-manager-65cfbc7dbc-cwswd\" (UID: \"b965c3f1-ed80-4b1f-8448-3df5096345e0\") " pod="openshift-lws-operator/lws-controller-manager-65cfbc7dbc-cwswd" Jul 08 03:05:57.081350 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:57.081307 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b965c3f1-ed80-4b1f-8448-3df5096345e0-cert\") pod \"lws-controller-manager-65cfbc7dbc-cwswd\" (UID: \"b965c3f1-ed80-4b1f-8448-3df5096345e0\") " pod="openshift-lws-operator/lws-controller-manager-65cfbc7dbc-cwswd" Jul 08 03:05:57.090768 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:57.090739 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9pn9\" (UniqueName: \"kubernetes.io/projected/b965c3f1-ed80-4b1f-8448-3df5096345e0-kube-api-access-t9pn9\") pod \"lws-controller-manager-65cfbc7dbc-cwswd\" (UID: \"b965c3f1-ed80-4b1f-8448-3df5096345e0\") " pod="openshift-lws-operator/lws-controller-manager-65cfbc7dbc-cwswd" Jul 08 03:05:57.138889 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:57.138843 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-65cfbc7dbc-cwswd" Jul 08 03:05:57.279476 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:57.279444 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-65cfbc7dbc-cwswd"] Jul 08 03:05:57.282262 ip-10-0-128-70 kubenswrapper[2564]: W0708 03:05:57.282239 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb965c3f1_ed80_4b1f_8448_3df5096345e0.slice/crio-67a31c8e47802ee75df181dc6dfe4c165291424447e05ddeacff876be3fe09f9 WatchSource:0}: Error finding container 67a31c8e47802ee75df181dc6dfe4c165291424447e05ddeacff876be3fe09f9: Status 404 returned error can't find the container with id 67a31c8e47802ee75df181dc6dfe4c165291424447e05ddeacff876be3fe09f9 Jul 08 03:05:57.619243 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:57.619204 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-65cfbc7dbc-cwswd" event={"ID":"b965c3f1-ed80-4b1f-8448-3df5096345e0","Type":"ContainerStarted","Data":"67a31c8e47802ee75df181dc6dfe4c165291424447e05ddeacff876be3fe09f9"} Jul 08 03:05:58.198905 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:58.198870 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:58.467354 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:58.467259 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:59.530757 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:59.530725 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:59.629882 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:59.629844 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-65cfbc7dbc-cwswd" event={"ID":"b965c3f1-ed80-4b1f-8448-3df5096345e0","Type":"ContainerStarted","Data":"7f630399759959c2464a1ebf7817dccd5e5078bf8200e99b26886a15d58ddc4e"} Jul 08 03:05:59.630067 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:59.629914 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-65cfbc7dbc-cwswd" Jul 08 03:05:59.664669 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:59.664608 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-65cfbc7dbc-cwswd" podStartSLOduration=2.319107899 podStartE2EDuration="3.664593598s" totalImagesPullingTime="1.345485699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:05:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:05:59.662778804 +0000 UTC m=+434.320154674" watchObservedRunningTime="2026-07-08 03:05:59.664593598 +0000 UTC m=+434.321969514" Jul 08 03:05:59.715020 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:59.714987 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:05:59.886625 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:05:59.886605 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:06:00.245587 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:00.245504 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:06:00.336007 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:00.335970 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-58b85c4fb4-cvslw"] Jul 08 03:06:00.338251 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:00.338229 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58b85c4fb4-cvslw" Jul 08 03:06:00.342071 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:00.342051 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 08 03:06:00.342166 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:00.342115 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 08 03:06:00.342457 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:00.342440 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 08 03:06:00.342882 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:00.342860 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 08 03:06:00.343309 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:00.343295 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-f2dvz\"" Jul 08 03:06:00.352230 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:00.352208 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58b85c4fb4-cvslw"] Jul 08 03:06:00.438468 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:00.438433 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:06:00.511056 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:00.510957 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/076a45d6-5c31-4e82-ac94-84aafddcdf69-tls-certs\") pod \"kube-auth-proxy-58b85c4fb4-cvslw\" (UID: \"076a45d6-5c31-4e82-ac94-84aafddcdf69\") " pod="openshift-ingress/kube-auth-proxy-58b85c4fb4-cvslw" Jul 08 03:06:00.511056 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:00.511007 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/076a45d6-5c31-4e82-ac94-84aafddcdf69-tmp\") pod \"kube-auth-proxy-58b85c4fb4-cvslw\" (UID: \"076a45d6-5c31-4e82-ac94-84aafddcdf69\") " pod="openshift-ingress/kube-auth-proxy-58b85c4fb4-cvslw" Jul 08 03:06:00.511056 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:00.511042 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mkg9\" (UniqueName: \"kubernetes.io/projected/076a45d6-5c31-4e82-ac94-84aafddcdf69-kube-api-access-7mkg9\") pod \"kube-auth-proxy-58b85c4fb4-cvslw\" (UID: \"076a45d6-5c31-4e82-ac94-84aafddcdf69\") " pod="openshift-ingress/kube-auth-proxy-58b85c4fb4-cvslw" Jul 08 03:06:00.611802 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:00.611756 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/076a45d6-5c31-4e82-ac94-84aafddcdf69-tls-certs\") pod \"kube-auth-proxy-58b85c4fb4-cvslw\" (UID: \"076a45d6-5c31-4e82-ac94-84aafddcdf69\") " pod="openshift-ingress/kube-auth-proxy-58b85c4fb4-cvslw" Jul 08 03:06:00.611802 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:00.611806 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/076a45d6-5c31-4e82-ac94-84aafddcdf69-tmp\") pod \"kube-auth-proxy-58b85c4fb4-cvslw\" (UID: \"076a45d6-5c31-4e82-ac94-84aafddcdf69\") " pod="openshift-ingress/kube-auth-proxy-58b85c4fb4-cvslw" Jul 08 03:06:00.612402 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:00.611831 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mkg9\" (UniqueName: \"kubernetes.io/projected/076a45d6-5c31-4e82-ac94-84aafddcdf69-kube-api-access-7mkg9\") pod \"kube-auth-proxy-58b85c4fb4-cvslw\" (UID: \"076a45d6-5c31-4e82-ac94-84aafddcdf69\") " pod="openshift-ingress/kube-auth-proxy-58b85c4fb4-cvslw" Jul 08 03:06:00.614777 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:00.614747 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/076a45d6-5c31-4e82-ac94-84aafddcdf69-tmp\") pod \"kube-auth-proxy-58b85c4fb4-cvslw\" (UID: \"076a45d6-5c31-4e82-ac94-84aafddcdf69\") " pod="openshift-ingress/kube-auth-proxy-58b85c4fb4-cvslw" Jul 08 03:06:00.614972 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:00.614839 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/076a45d6-5c31-4e82-ac94-84aafddcdf69-tls-certs\") pod \"kube-auth-proxy-58b85c4fb4-cvslw\" (UID: \"076a45d6-5c31-4e82-ac94-84aafddcdf69\") " pod="openshift-ingress/kube-auth-proxy-58b85c4fb4-cvslw" Jul 08 03:06:00.628569 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:00.628534 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mkg9\" (UniqueName: \"kubernetes.io/projected/076a45d6-5c31-4e82-ac94-84aafddcdf69-kube-api-access-7mkg9\") pod \"kube-auth-proxy-58b85c4fb4-cvslw\" (UID: \"076a45d6-5c31-4e82-ac94-84aafddcdf69\") " pod="openshift-ingress/kube-auth-proxy-58b85c4fb4-cvslw" Jul 08 03:06:00.647880 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:00.647848 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58b85c4fb4-cvslw" Jul 08 03:06:00.802160 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:00.802079 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:06:00.822677 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:00.822656 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58b85c4fb4-cvslw"] Jul 08 03:06:00.824385 ip-10-0-128-70 kubenswrapper[2564]: W0708 03:06:00.824359 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076a45d6_5c31_4e82_ac94_84aafddcdf69.slice/crio-af923fb115bcc28065676d3d5aff2b526c25c6ab7a6203af273d17126cd41a3f WatchSource:0}: Error finding container af923fb115bcc28065676d3d5aff2b526c25c6ab7a6203af273d17126cd41a3f: Status 404 returned error can't find the container with id af923fb115bcc28065676d3d5aff2b526c25c6ab7a6203af273d17126cd41a3f Jul 08 03:06:01.172640 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:01.172598 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:06:01.640597 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:01.640556 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58b85c4fb4-cvslw" event={"ID":"076a45d6-5c31-4e82-ac94-84aafddcdf69","Type":"ContainerStarted","Data":"af923fb115bcc28065676d3d5aff2b526c25c6ab7a6203af273d17126cd41a3f"} Jul 08 03:06:01.977371 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:01.977227 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:06:02.070464 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:02.070431 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:06:03.150592 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:03.150553 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:06:04.658979 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:04.658947 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58b85c4fb4-cvslw" event={"ID":"076a45d6-5c31-4e82-ac94-84aafddcdf69","Type":"ContainerStarted","Data":"22a1eb3b4a1266f3534a2884daca743579e2bacd58f82e483a49219d313da2e5"} Jul 08 03:06:04.676070 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:04.676019 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-58b85c4fb4-cvslw" podStartSLOduration=1.585653696 podStartE2EDuration="4.676002926s" totalImagesPullingTime="3.09034923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:06:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:06:04.674437686 +0000 UTC m=+439.331813571" watchObservedRunningTime="2026-07-08 03:06:04.676002926 +0000 UTC m=+439.333378797" Jul 08 03:06:08.193523 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:08.193430 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:06:08.296307 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:08.296274 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:06:09.121430 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.121399 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r"] Jul 08 03:06:09.124520 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.124502 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.128377 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.128355 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 03:06:09.128496 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.128399 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-cnshq\"" Jul 08 03:06:09.139234 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.139206 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r"] Jul 08 03:06:09.145910 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.145878 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:06:09.183407 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.183370 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059975b3-ece8-476b-a559-45b6d1c2bf95-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vhx4r\" (UID: \"059975b3-ece8-476b-a559-45b6d1c2bf95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.183407 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.183408 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059975b3-ece8-476b-a559-45b6d1c2bf95-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vhx4r\" (UID: \"059975b3-ece8-476b-a559-45b6d1c2bf95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.183665 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.183425 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059975b3-ece8-476b-a559-45b6d1c2bf95-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vhx4r\" (UID: \"059975b3-ece8-476b-a559-45b6d1c2bf95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.183665 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.183487 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059975b3-ece8-476b-a559-45b6d1c2bf95-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vhx4r\" (UID: \"059975b3-ece8-476b-a559-45b6d1c2bf95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.183665 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.183522 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059975b3-ece8-476b-a559-45b6d1c2bf95-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vhx4r\" (UID: \"059975b3-ece8-476b-a559-45b6d1c2bf95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.183665 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.183548 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059975b3-ece8-476b-a559-45b6d1c2bf95-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vhx4r\" (UID: \"059975b3-ece8-476b-a559-45b6d1c2bf95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.183665 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.183593 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn5zq\" (UniqueName: \"kubernetes.io/projected/059975b3-ece8-476b-a559-45b6d1c2bf95-kube-api-access-qn5zq\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vhx4r\" (UID: \"059975b3-ece8-476b-a559-45b6d1c2bf95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.183665 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.183629 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059975b3-ece8-476b-a559-45b6d1c2bf95-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vhx4r\" (UID: \"059975b3-ece8-476b-a559-45b6d1c2bf95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.183665 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.183645 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059975b3-ece8-476b-a559-45b6d1c2bf95-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vhx4r\" (UID: \"059975b3-ece8-476b-a559-45b6d1c2bf95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.284531 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.284496 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059975b3-ece8-476b-a559-45b6d1c2bf95-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vhx4r\" (UID: \"059975b3-ece8-476b-a559-45b6d1c2bf95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.284985 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.284557 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059975b3-ece8-476b-a559-45b6d1c2bf95-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vhx4r\" (UID: \"059975b3-ece8-476b-a559-45b6d1c2bf95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.284985 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.284580 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059975b3-ece8-476b-a559-45b6d1c2bf95-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vhx4r\" (UID: \"059975b3-ece8-476b-a559-45b6d1c2bf95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.284985 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.284595 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059975b3-ece8-476b-a559-45b6d1c2bf95-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vhx4r\" (UID: \"059975b3-ece8-476b-a559-45b6d1c2bf95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.284985 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.284612 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059975b3-ece8-476b-a559-45b6d1c2bf95-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vhx4r\" (UID: \"059975b3-ece8-476b-a559-45b6d1c2bf95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.284985 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.284628 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059975b3-ece8-476b-a559-45b6d1c2bf95-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vhx4r\" (UID: \"059975b3-ece8-476b-a559-45b6d1c2bf95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.284985 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.284643 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059975b3-ece8-476b-a559-45b6d1c2bf95-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vhx4r\" (UID: \"059975b3-ece8-476b-a559-45b6d1c2bf95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.284985 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.284664 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn5zq\" (UniqueName: \"kubernetes.io/projected/059975b3-ece8-476b-a559-45b6d1c2bf95-kube-api-access-qn5zq\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vhx4r\" (UID: \"059975b3-ece8-476b-a559-45b6d1c2bf95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.284985 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.284716 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059975b3-ece8-476b-a559-45b6d1c2bf95-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vhx4r\" (UID: \"059975b3-ece8-476b-a559-45b6d1c2bf95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.285378 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.285153 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/059975b3-ece8-476b-a559-45b6d1c2bf95-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vhx4r\" (UID: \"059975b3-ece8-476b-a559-45b6d1c2bf95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.285378 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.285149 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/059975b3-ece8-476b-a559-45b6d1c2bf95-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vhx4r\" (UID: \"059975b3-ece8-476b-a559-45b6d1c2bf95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.285378 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.285179 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/059975b3-ece8-476b-a559-45b6d1c2bf95-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vhx4r\" (UID: \"059975b3-ece8-476b-a559-45b6d1c2bf95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.285524 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.285477 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/059975b3-ece8-476b-a559-45b6d1c2bf95-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vhx4r\" (UID: \"059975b3-ece8-476b-a559-45b6d1c2bf95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.285577 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.285566 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/059975b3-ece8-476b-a559-45b6d1c2bf95-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vhx4r\" (UID: \"059975b3-ece8-476b-a559-45b6d1c2bf95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.287042 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.287022 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/059975b3-ece8-476b-a559-45b6d1c2bf95-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vhx4r\" (UID: \"059975b3-ece8-476b-a559-45b6d1c2bf95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.287275 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.287258 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/059975b3-ece8-476b-a559-45b6d1c2bf95-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vhx4r\" (UID: \"059975b3-ece8-476b-a559-45b6d1c2bf95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.306386 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.306358 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/059975b3-ece8-476b-a559-45b6d1c2bf95-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vhx4r\" (UID: \"059975b3-ece8-476b-a559-45b6d1c2bf95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.332048 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.332020 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn5zq\" (UniqueName: \"kubernetes.io/projected/059975b3-ece8-476b-a559-45b6d1c2bf95-kube-api-access-qn5zq\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vhx4r\" (UID: \"059975b3-ece8-476b-a559-45b6d1c2bf95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.434489 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.434443 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:09.469064 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.468989 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:06:09.599240 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.599211 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:06:09.615682 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.615653 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r"] Jul 08 03:06:09.617282 ip-10-0-128-70 kubenswrapper[2564]: W0708 03:06:09.617239 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod059975b3_ece8_476b_a559_45b6d1c2bf95.slice/crio-d243e3d8b603ffdd26e21f610a37e8dd34b3e6034519e8ee376b4fb15e8749e8 WatchSource:0}: Error finding container d243e3d8b603ffdd26e21f610a37e8dd34b3e6034519e8ee376b4fb15e8749e8: Status 404 returned error can't find the container with id d243e3d8b603ffdd26e21f610a37e8dd34b3e6034519e8ee376b4fb15e8749e8 Jul 08 03:06:09.680982 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:09.680934 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" event={"ID":"059975b3-ece8-476b-a559-45b6d1c2bf95","Type":"ContainerStarted","Data":"d243e3d8b603ffdd26e21f610a37e8dd34b3e6034519e8ee376b4fb15e8749e8"} Jul 08 03:06:10.637421 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:10.637389 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-65cfbc7dbc-cwswd" Jul 08 03:06:11.375377 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:11.374985 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:06:11.449489 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:11.449456 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-2hkzh"] Jul 08 03:06:11.453253 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:11.453226 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2hkzh" Jul 08 03:06:11.456267 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:11.456243 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-8p62f\"" Jul 08 03:06:11.456396 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:11.456297 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 08 03:06:11.470027 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:11.469994 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-2hkzh"] Jul 08 03:06:11.484098 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:11.484065 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:06:11.507905 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:11.507874 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtgj5\" (UniqueName: \"kubernetes.io/projected/a8816a59-d5ac-41c4-b35a-c7a914e61ac7-kube-api-access-dtgj5\") pod \"llmisvc-controller-manager-777c6bd7d5-2hkzh\" (UID: \"a8816a59-d5ac-41c4-b35a-c7a914e61ac7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2hkzh" Jul 08 03:06:11.508063 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:11.507931 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8816a59-d5ac-41c4-b35a-c7a914e61ac7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-2hkzh\" (UID: \"a8816a59-d5ac-41c4-b35a-c7a914e61ac7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2hkzh" Jul 08 03:06:11.607076 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:11.607042 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:06:11.609185 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:11.609164 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtgj5\" (UniqueName: \"kubernetes.io/projected/a8816a59-d5ac-41c4-b35a-c7a914e61ac7-kube-api-access-dtgj5\") pod \"llmisvc-controller-manager-777c6bd7d5-2hkzh\" (UID: \"a8816a59-d5ac-41c4-b35a-c7a914e61ac7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2hkzh" Jul 08 03:06:11.609316 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:11.609199 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8816a59-d5ac-41c4-b35a-c7a914e61ac7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-2hkzh\" (UID: \"a8816a59-d5ac-41c4-b35a-c7a914e61ac7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2hkzh" Jul 08 03:06:11.612046 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:11.612022 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8816a59-d5ac-41c4-b35a-c7a914e61ac7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-2hkzh\" (UID: \"a8816a59-d5ac-41c4-b35a-c7a914e61ac7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2hkzh" Jul 08 03:06:11.618953 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:11.618865 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtgj5\" (UniqueName: \"kubernetes.io/projected/a8816a59-d5ac-41c4-b35a-c7a914e61ac7-kube-api-access-dtgj5\") pod \"llmisvc-controller-manager-777c6bd7d5-2hkzh\" (UID: \"a8816a59-d5ac-41c4-b35a-c7a914e61ac7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2hkzh" Jul 08 03:06:11.767064 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:11.767022 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2hkzh" Jul 08 03:06:11.885789 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:11.885756 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:06:12.543354 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:12.543315 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:06:12.543459 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:12.543395 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:06:12.543459 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:12.543433 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:06:12.670821 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:12.670782 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-2hkzh"] Jul 08 03:06:12.675694 ip-10-0-128-70 kubenswrapper[2564]: W0708 03:06:12.675663 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda8816a59_d5ac_41c4_b35a_c7a914e61ac7.slice/crio-1bc534ad8eea4650ed7f26ca4c1ae3307305d9d20a0125382517fe93776da2fb WatchSource:0}: Error finding container 1bc534ad8eea4650ed7f26ca4c1ae3307305d9d20a0125382517fe93776da2fb: Status 404 returned error can't find the container with id 1bc534ad8eea4650ed7f26ca4c1ae3307305d9d20a0125382517fe93776da2fb Jul 08 03:06:12.701071 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:12.701033 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" event={"ID":"059975b3-ece8-476b-a559-45b6d1c2bf95","Type":"ContainerStarted","Data":"6918bce2c74c0cd205425bfef5dca46b5b39993dbd565594d81f17970caac7eb"} Jul 08 03:06:12.702416 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:12.702392 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2hkzh" event={"ID":"a8816a59-d5ac-41c4-b35a-c7a914e61ac7","Type":"ContainerStarted","Data":"1bc534ad8eea4650ed7f26ca4c1ae3307305d9d20a0125382517fe93776da2fb"} Jul 08 03:06:12.748785 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:12.748686 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" podStartSLOduration=0.8252515 podStartE2EDuration="3.748668315s" totalImagesPullingTime="2.923416815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:06:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:06:12.74669653 +0000 UTC m=+447.404072426" watchObservedRunningTime="2026-07-08 03:06:12.748668315 +0000 UTC m=+447.406044185" Jul 08 03:06:12.770963 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:12.770907 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:06:12.915580 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:12.915547 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:06:13.014232 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:13.014138 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:06:13.434902 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:13.434865 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:13.439493 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:13.439463 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:13.708055 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:13.707972 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:13.709037 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:13.709018 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vhx4r" Jul 08 03:06:16.723147 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:16.723107 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2hkzh" event={"ID":"a8816a59-d5ac-41c4-b35a-c7a914e61ac7","Type":"ContainerStarted","Data":"d58a7edbe88dc3c7d55fd9d4f1b687e410bcef4f3ed93b0b040e65f9f9d06df0"} Jul 08 03:06:16.723556 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:16.723191 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2hkzh" Jul 08 03:06:16.742361 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:16.742302 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2hkzh" podStartSLOduration=2.131894452 podStartE2EDuration="5.742283176s" totalImagesPullingTime="3.610388724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:06:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:06:16.74155994 +0000 UTC m=+451.398935811" watchObservedRunningTime="2026-07-08 03:06:16.742283176 +0000 UTC m=+451.399659047" Jul 08 03:06:18.259783 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:18.259733 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:06:21.926631 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:21.926576 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:06:24.075781 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:24.075745 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:06:24.175674 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:24.175638 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:06:24.249438 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:24.249402 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:06:47.729841 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:47.729810 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2hkzh" Jul 08 03:06:59.767513 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:59.767470 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:06:59.856175 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:59.856120 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:06:59.944243 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:06:59.944207 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:07:03.723568 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:03.723537 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:07:28.840985 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:28.840952 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:07:32.457431 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:32.457394 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:07:32.570808 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:32.570770 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:07:37.468706 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:37.468617 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:07:37.706164 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:37.706116 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:07:39.904313 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:39.904261 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:07:40.097385 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:40.097349 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:07:40.241971 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:40.241817 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:07:42.476770 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:42.476725 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:07:44.453145 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:44.453111 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:07:44.555762 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:44.555711 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:07:46.325433 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:46.325396 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-p7fjt"] Jul 08 03:07:46.328825 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:46.328804 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p7fjt" Jul 08 03:07:46.331770 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:46.331750 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 08 03:07:46.331983 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:46.331966 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 08 03:07:46.332353 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:46.332333 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 08 03:07:46.333057 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:46.333041 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 08 03:07:46.333165 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:46.333148 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hqzwg\"" Jul 08 03:07:46.341622 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:46.341549 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:07:46.343837 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:46.343813 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-p7fjt"] Jul 08 03:07:46.464755 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:46.464715 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5f5ba59-ab30-4609-9c91-4e205ee9710b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-p7fjt\" (UID: \"b5f5ba59-ab30-4609-9c91-4e205ee9710b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p7fjt" Jul 08 03:07:46.464755 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:46.464755 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b5f5ba59-ab30-4609-9c91-4e205ee9710b-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-p7fjt\" (UID: \"b5f5ba59-ab30-4609-9c91-4e205ee9710b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p7fjt" Jul 08 03:07:46.464996 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:46.464772 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt4fm\" (UniqueName: \"kubernetes.io/projected/b5f5ba59-ab30-4609-9c91-4e205ee9710b-kube-api-access-wt4fm\") pod \"kuadrant-console-plugin-66b5774985-p7fjt\" (UID: \"b5f5ba59-ab30-4609-9c91-4e205ee9710b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p7fjt" Jul 08 03:07:46.565706 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:46.565672 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5f5ba59-ab30-4609-9c91-4e205ee9710b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-p7fjt\" (UID: \"b5f5ba59-ab30-4609-9c91-4e205ee9710b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p7fjt" Jul 08 03:07:46.565706 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:46.565706 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b5f5ba59-ab30-4609-9c91-4e205ee9710b-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-p7fjt\" (UID: \"b5f5ba59-ab30-4609-9c91-4e205ee9710b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p7fjt" Jul 08 03:07:46.565905 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:46.565730 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt4fm\" (UniqueName: \"kubernetes.io/projected/b5f5ba59-ab30-4609-9c91-4e205ee9710b-kube-api-access-wt4fm\") pod \"kuadrant-console-plugin-66b5774985-p7fjt\" (UID: \"b5f5ba59-ab30-4609-9c91-4e205ee9710b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p7fjt" Jul 08 03:07:46.565905 ip-10-0-128-70 kubenswrapper[2564]: E0708 03:07:46.565844 2564 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 08 03:07:46.565990 ip-10-0-128-70 kubenswrapper[2564]: E0708 03:07:46.565939 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5f5ba59-ab30-4609-9c91-4e205ee9710b-plugin-serving-cert podName:b5f5ba59-ab30-4609-9c91-4e205ee9710b nodeName:}" failed. No retries permitted until 2026-07-08 03:07:47.065904552 +0000 UTC m=+541.723280402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/b5f5ba59-ab30-4609-9c91-4e205ee9710b-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-p7fjt" (UID: "b5f5ba59-ab30-4609-9c91-4e205ee9710b") : secret "plugin-serving-cert" not found Jul 08 03:07:46.566374 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:46.566354 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b5f5ba59-ab30-4609-9c91-4e205ee9710b-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-p7fjt\" (UID: \"b5f5ba59-ab30-4609-9c91-4e205ee9710b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p7fjt" Jul 08 03:07:46.575478 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:46.575416 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt4fm\" (UniqueName: \"kubernetes.io/projected/b5f5ba59-ab30-4609-9c91-4e205ee9710b-kube-api-access-wt4fm\") pod \"kuadrant-console-plugin-66b5774985-p7fjt\" (UID: \"b5f5ba59-ab30-4609-9c91-4e205ee9710b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p7fjt" Jul 08 03:07:46.627235 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:46.627190 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:07:47.070784 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:47.070743 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5f5ba59-ab30-4609-9c91-4e205ee9710b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-p7fjt\" (UID: \"b5f5ba59-ab30-4609-9c91-4e205ee9710b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p7fjt" Jul 08 03:07:47.073318 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:47.073298 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5f5ba59-ab30-4609-9c91-4e205ee9710b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-p7fjt\" (UID: \"b5f5ba59-ab30-4609-9c91-4e205ee9710b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p7fjt" Jul 08 03:07:47.237698 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:47.237668 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p7fjt" Jul 08 03:07:47.375008 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:47.374984 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-p7fjt"] Jul 08 03:07:47.377192 ip-10-0-128-70 kubenswrapper[2564]: W0708 03:07:47.377162 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5f5ba59_ab30_4609_9c91_4e205ee9710b.slice/crio-ce413f4ff6750d0494c5809fd9cbc90b1c84b9c88af1752262d1458ea95101cf WatchSource:0}: Error finding container ce413f4ff6750d0494c5809fd9cbc90b1c84b9c88af1752262d1458ea95101cf: Status 404 returned error can't find the container with id ce413f4ff6750d0494c5809fd9cbc90b1c84b9c88af1752262d1458ea95101cf Jul 08 03:07:47.657964 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:47.657911 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:07:47.760876 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:47.760841 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:07:48.121780 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:48.121696 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p7fjt" event={"ID":"b5f5ba59-ab30-4609-9c91-4e205ee9710b","Type":"ContainerStarted","Data":"ce413f4ff6750d0494c5809fd9cbc90b1c84b9c88af1752262d1458ea95101cf"} Jul 08 03:07:53.917479 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:53.917435 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:07:54.019615 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:54.019580 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:07:56.674537 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:07:56.674497 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:08:00.467466 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:00.467425 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:08:00.568574 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:00.568542 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:08:03.085140 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:03.085100 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:08:03.183253 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:03.183213 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:08:06.066909 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:06.066874 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:08:06.129905 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:06.129867 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:08:11.237353 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:11.237313 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p7fjt" event={"ID":"b5f5ba59-ab30-4609-9c91-4e205ee9710b","Type":"ContainerStarted","Data":"8b4db3af7a0ff9fff7e135a6fd96985924fdaf15820df01c5ef439157ac3422a"} Jul 08 03:08:11.255267 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:11.255213 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p7fjt" podStartSLOduration=1.589117482 podStartE2EDuration="25.255197068s" totalImagesPullingTime="23.666079586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:08:11.254134848 +0000 UTC m=+565.911510718" watchObservedRunningTime="2026-07-08 03:08:11.255197068 +0000 UTC m=+565.912572939" Jul 08 03:08:12.984966 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:12.984908 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:08:13.040072 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:13.040039 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:08:13.661771 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:13.661736 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:08:13.717590 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:13.717561 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:08:19.692440 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:19.692400 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:08:19.759636 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:19.759596 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:08:19.861204 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:19.861171 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:08:19.961350 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:19.961265 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:08:24.915907 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:24.915875 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7"] Jul 08 03:08:24.939682 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:24.939639 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:08:24.965090 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:24.965054 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7"] Jul 08 03:08:24.965247 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:24.965153 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:24.968254 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:24.968230 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-wh4d7\"" Jul 08 03:08:25.015307 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.015214 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/631c38ff-a249-470e-b1c5-bf4f761e4b71-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-lzlg7\" (UID: \"631c38ff-a249-470e-b1c5-bf4f761e4b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:25.015307 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.015302 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/631c38ff-a249-470e-b1c5-bf4f761e4b71-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-lzlg7\" (UID: \"631c38ff-a249-470e-b1c5-bf4f761e4b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:25.015571 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.015362 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/631c38ff-a249-470e-b1c5-bf4f761e4b71-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-lzlg7\" (UID: \"631c38ff-a249-470e-b1c5-bf4f761e4b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:25.015571 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.015392 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/631c38ff-a249-470e-b1c5-bf4f761e4b71-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lzlg7\" (UID: \"631c38ff-a249-470e-b1c5-bf4f761e4b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:25.015571 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.015424 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/631c38ff-a249-470e-b1c5-bf4f761e4b71-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-lzlg7\" (UID: \"631c38ff-a249-470e-b1c5-bf4f761e4b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:25.015571 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.015507 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/631c38ff-a249-470e-b1c5-bf4f761e4b71-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lzlg7\" (UID: \"631c38ff-a249-470e-b1c5-bf4f761e4b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:25.015571 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.015550 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/631c38ff-a249-470e-b1c5-bf4f761e4b71-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-lzlg7\" (UID: \"631c38ff-a249-470e-b1c5-bf4f761e4b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:25.015830 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.015582 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/631c38ff-a249-470e-b1c5-bf4f761e4b71-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-lzlg7\" (UID: \"631c38ff-a249-470e-b1c5-bf4f761e4b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:25.015830 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.015627 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqfzp\" (UniqueName: \"kubernetes.io/projected/631c38ff-a249-470e-b1c5-bf4f761e4b71-kube-api-access-gqfzp\") pod \"maas-default-gateway-openshift-default-56654dcf7-lzlg7\" (UID: \"631c38ff-a249-470e-b1c5-bf4f761e4b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:25.116520 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.116488 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/631c38ff-a249-470e-b1c5-bf4f761e4b71-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lzlg7\" (UID: \"631c38ff-a249-470e-b1c5-bf4f761e4b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:25.116714 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.116536 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/631c38ff-a249-470e-b1c5-bf4f761e4b71-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-lzlg7\" (UID: \"631c38ff-a249-470e-b1c5-bf4f761e4b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:25.116714 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.116569 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/631c38ff-a249-470e-b1c5-bf4f761e4b71-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-lzlg7\" (UID: \"631c38ff-a249-470e-b1c5-bf4f761e4b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:25.116714 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.116599 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqfzp\" (UniqueName: \"kubernetes.io/projected/631c38ff-a249-470e-b1c5-bf4f761e4b71-kube-api-access-gqfzp\") pod \"maas-default-gateway-openshift-default-56654dcf7-lzlg7\" (UID: \"631c38ff-a249-470e-b1c5-bf4f761e4b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:25.116714 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.116635 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/631c38ff-a249-470e-b1c5-bf4f761e4b71-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-lzlg7\" (UID: \"631c38ff-a249-470e-b1c5-bf4f761e4b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:25.116714 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.116689 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/631c38ff-a249-470e-b1c5-bf4f761e4b71-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-lzlg7\" (UID: \"631c38ff-a249-470e-b1c5-bf4f761e4b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:25.116994 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.116730 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/631c38ff-a249-470e-b1c5-bf4f761e4b71-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-lzlg7\" (UID: \"631c38ff-a249-470e-b1c5-bf4f761e4b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:25.116994 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.116759 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/631c38ff-a249-470e-b1c5-bf4f761e4b71-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lzlg7\" (UID: \"631c38ff-a249-470e-b1c5-bf4f761e4b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:25.116994 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.116791 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/631c38ff-a249-470e-b1c5-bf4f761e4b71-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-lzlg7\" (UID: \"631c38ff-a249-470e-b1c5-bf4f761e4b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:25.116994 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.116962 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/631c38ff-a249-470e-b1c5-bf4f761e4b71-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lzlg7\" (UID: \"631c38ff-a249-470e-b1c5-bf4f761e4b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:25.117193 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.117107 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/631c38ff-a249-470e-b1c5-bf4f761e4b71-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-lzlg7\" (UID: \"631c38ff-a249-470e-b1c5-bf4f761e4b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:25.117249 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.117208 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/631c38ff-a249-470e-b1c5-bf4f761e4b71-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-lzlg7\" (UID: \"631c38ff-a249-470e-b1c5-bf4f761e4b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:25.117298 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.117265 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/631c38ff-a249-470e-b1c5-bf4f761e4b71-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lzlg7\" (UID: \"631c38ff-a249-470e-b1c5-bf4f761e4b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:25.117505 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.117478 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/631c38ff-a249-470e-b1c5-bf4f761e4b71-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-lzlg7\" (UID: \"631c38ff-a249-470e-b1c5-bf4f761e4b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:25.119221 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.119196 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/631c38ff-a249-470e-b1c5-bf4f761e4b71-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-lzlg7\" (UID: \"631c38ff-a249-470e-b1c5-bf4f761e4b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:25.119337 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.119318 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/631c38ff-a249-470e-b1c5-bf4f761e4b71-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-lzlg7\" (UID: \"631c38ff-a249-470e-b1c5-bf4f761e4b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:25.129775 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.129749 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/631c38ff-a249-470e-b1c5-bf4f761e4b71-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-lzlg7\" (UID: \"631c38ff-a249-470e-b1c5-bf4f761e4b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:25.129958 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.129888 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqfzp\" (UniqueName: \"kubernetes.io/projected/631c38ff-a249-470e-b1c5-bf4f761e4b71-kube-api-access-gqfzp\") pod \"maas-default-gateway-openshift-default-56654dcf7-lzlg7\" (UID: \"631c38ff-a249-470e-b1c5-bf4f761e4b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:25.275063 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.274967 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:25.294777 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.294727 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:08:25.413557 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.413521 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7"] Jul 08 03:08:25.416484 ip-10-0-128-70 kubenswrapper[2564]: W0708 03:08:25.416461 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod631c38ff_a249_470e_b1c5_bf4f761e4b71.slice/crio-050ed51971388778eb95ea5bf4bbd61d5f7ed0a80088f8bb8c7924c0ccd1ffdf WatchSource:0}: Error finding container 050ed51971388778eb95ea5bf4bbd61d5f7ed0a80088f8bb8c7924c0ccd1ffdf: Status 404 returned error can't find the container with id 050ed51971388778eb95ea5bf4bbd61d5f7ed0a80088f8bb8c7924c0ccd1ffdf Jul 08 03:08:25.418642 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.418608 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:08:25.418730 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.418680 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:08:25.418730 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:25.418710 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:08:26.303771 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:26.303726 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" event={"ID":"631c38ff-a249-470e-b1c5-bf4f761e4b71","Type":"ContainerStarted","Data":"7cf581d45abfc4cd24e9be816455af990d628df0e3eb748a273ad95d53ed558d"} Jul 08 03:08:26.303771 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:26.303776 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" event={"ID":"631c38ff-a249-470e-b1c5-bf4f761e4b71","Type":"ContainerStarted","Data":"050ed51971388778eb95ea5bf4bbd61d5f7ed0a80088f8bb8c7924c0ccd1ffdf"} Jul 08 03:08:26.326312 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:26.326256 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" podStartSLOduration=2.32624052 podStartE2EDuration="2.32624052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:08:26.324571233 +0000 UTC m=+580.981947105" watchObservedRunningTime="2026-07-08 03:08:26.32624052 +0000 UTC m=+580.983616390" Jul 08 03:08:27.275125 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:27.275084 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:27.280253 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:27.280226 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:27.307744 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:27.307709 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:27.308589 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:27.308574 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:08:27.388054 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:27.388020 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:08:27.497743 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:27.497710 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:08:27.601793 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:27.601676 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:08:28.188177 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:28.188143 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-c5xlh"] Jul 08 03:08:28.191966 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:28.191943 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-c5xlh" Jul 08 03:08:28.194533 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:28.194506 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pptkn\"" Jul 08 03:08:28.199295 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:28.199274 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-c5xlh"] Jul 08 03:08:28.227478 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:28.227441 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:08:28.245690 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:28.245658 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h6gf\" (UniqueName: \"kubernetes.io/projected/0fad0396-3157-4f5e-8f4d-19d383c0308b-kube-api-access-8h6gf\") pod \"authorino-58cd955675-c5xlh\" (UID: \"0fad0396-3157-4f5e-8f4d-19d383c0308b\") " pod="kuadrant-system/authorino-58cd955675-c5xlh" Jul 08 03:08:28.288789 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:28.288754 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:08:28.347098 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:28.347061 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h6gf\" (UniqueName: \"kubernetes.io/projected/0fad0396-3157-4f5e-8f4d-19d383c0308b-kube-api-access-8h6gf\") pod \"authorino-58cd955675-c5xlh\" (UID: \"0fad0396-3157-4f5e-8f4d-19d383c0308b\") " pod="kuadrant-system/authorino-58cd955675-c5xlh" Jul 08 03:08:28.356912 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:28.356880 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h6gf\" (UniqueName: \"kubernetes.io/projected/0fad0396-3157-4f5e-8f4d-19d383c0308b-kube-api-access-8h6gf\") pod \"authorino-58cd955675-c5xlh\" (UID: \"0fad0396-3157-4f5e-8f4d-19d383c0308b\") " pod="kuadrant-system/authorino-58cd955675-c5xlh" Jul 08 03:08:28.502336 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:28.502242 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-c5xlh" Jul 08 03:08:28.638426 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:28.638400 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-c5xlh"] Jul 08 03:08:28.640553 ip-10-0-128-70 kubenswrapper[2564]: W0708 03:08:28.640513 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fad0396_3157_4f5e_8f4d_19d383c0308b.slice/crio-e3f82c6fb944d2ff24cf0a0cb5a5e37012e530dec2efc5598961b1f7dc55cc00 WatchSource:0}: Error finding container e3f82c6fb944d2ff24cf0a0cb5a5e37012e530dec2efc5598961b1f7dc55cc00: Status 404 returned error can't find the container with id e3f82c6fb944d2ff24cf0a0cb5a5e37012e530dec2efc5598961b1f7dc55cc00 Jul 08 03:08:29.322026 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:29.321707 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-c5xlh" event={"ID":"0fad0396-3157-4f5e-8f4d-19d383c0308b","Type":"ContainerStarted","Data":"e3f82c6fb944d2ff24cf0a0cb5a5e37012e530dec2efc5598961b1f7dc55cc00"} Jul 08 03:08:32.339710 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:32.339615 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-c5xlh" event={"ID":"0fad0396-3157-4f5e-8f4d-19d383c0308b","Type":"ContainerStarted","Data":"3a40c77b553a4c575caea44db440a30b5c04d014de32a690d33bb4376d215cc7"} Jul 08 03:08:32.358118 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:32.358063 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-c5xlh" podStartSLOduration=1.039144136 podStartE2EDuration="4.358045524s" totalImagesPullingTime="3.318901388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:08:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:08:32.357531302 +0000 UTC m=+587.014907173" watchObservedRunningTime="2026-07-08 03:08:32.358045524 +0000 UTC m=+587.015421401" Jul 08 03:08:33.666174 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:33.666142 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-c5xlh"] Jul 08 03:08:34.348790 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:34.348742 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-c5xlh" podUID="0fad0396-3157-4f5e-8f4d-19d383c0308b" containerName="authorino" containerID="cri-o://3a40c77b553a4c575caea44db440a30b5c04d014de32a690d33bb4376d215cc7" gracePeriod=30 Jul 08 03:08:34.600679 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:34.600610 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-c5xlh" Jul 08 03:08:34.707714 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:34.707664 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8h6gf\" (UniqueName: \"kubernetes.io/projected/0fad0396-3157-4f5e-8f4d-19d383c0308b-kube-api-access-8h6gf\") pod \"0fad0396-3157-4f5e-8f4d-19d383c0308b\" (UID: \"0fad0396-3157-4f5e-8f4d-19d383c0308b\") " Jul 08 03:08:34.709972 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:34.709946 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fad0396-3157-4f5e-8f4d-19d383c0308b-kube-api-access-8h6gf" (OuterVolumeSpecName: "kube-api-access-8h6gf") pod "0fad0396-3157-4f5e-8f4d-19d383c0308b" (UID: "0fad0396-3157-4f5e-8f4d-19d383c0308b"). InnerVolumeSpecName "kube-api-access-8h6gf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:08:34.808526 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:34.808493 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h6gf\" (UniqueName: \"kubernetes.io/projected/0fad0396-3157-4f5e-8f4d-19d383c0308b-kube-api-access-8h6gf\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:08:35.354070 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:35.354039 2564 generic.go:356] "Generic (PLEG): container finished" podID="0fad0396-3157-4f5e-8f4d-19d383c0308b" containerID="3a40c77b553a4c575caea44db440a30b5c04d014de32a690d33bb4376d215cc7" exitCode=0 Jul 08 03:08:35.354259 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:35.354079 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-c5xlh" Jul 08 03:08:35.354259 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:35.354096 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-c5xlh" event={"ID":"0fad0396-3157-4f5e-8f4d-19d383c0308b","Type":"ContainerDied","Data":"3a40c77b553a4c575caea44db440a30b5c04d014de32a690d33bb4376d215cc7"} Jul 08 03:08:35.354259 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:35.354130 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-c5xlh" event={"ID":"0fad0396-3157-4f5e-8f4d-19d383c0308b","Type":"ContainerDied","Data":"e3f82c6fb944d2ff24cf0a0cb5a5e37012e530dec2efc5598961b1f7dc55cc00"} Jul 08 03:08:35.354259 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:35.354146 2564 scope.go:117] "RemoveContainer" containerID="3a40c77b553a4c575caea44db440a30b5c04d014de32a690d33bb4376d215cc7" Jul 08 03:08:35.363098 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:35.363082 2564 scope.go:117] "RemoveContainer" containerID="3a40c77b553a4c575caea44db440a30b5c04d014de32a690d33bb4376d215cc7" Jul 08 03:08:35.363392 ip-10-0-128-70 kubenswrapper[2564]: E0708 03:08:35.363373 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a40c77b553a4c575caea44db440a30b5c04d014de32a690d33bb4376d215cc7\": container with ID starting with 3a40c77b553a4c575caea44db440a30b5c04d014de32a690d33bb4376d215cc7 not found: ID does not exist" containerID="3a40c77b553a4c575caea44db440a30b5c04d014de32a690d33bb4376d215cc7" Jul 08 03:08:35.363430 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:35.363401 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a40c77b553a4c575caea44db440a30b5c04d014de32a690d33bb4376d215cc7"} err="failed to get container status \"3a40c77b553a4c575caea44db440a30b5c04d014de32a690d33bb4376d215cc7\": rpc error: code = NotFound desc = could not find container \"3a40c77b553a4c575caea44db440a30b5c04d014de32a690d33bb4376d215cc7\": container with ID starting with 3a40c77b553a4c575caea44db440a30b5c04d014de32a690d33bb4376d215cc7 not found: ID does not exist" Jul 08 03:08:35.374450 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:35.374423 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-c5xlh"] Jul 08 03:08:35.378100 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:35.378078 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-c5xlh"] Jul 08 03:08:35.885911 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:35.885882 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fad0396-3157-4f5e-8f4d-19d383c0308b" path="/var/lib/kubelet/pods/0fad0396-3157-4f5e-8f4d-19d383c0308b/volumes" Jul 08 03:08:48.534740 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:48.534708 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:08:48.632299 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:48.632264 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:08:48.991714 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:08:48.991671 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:02.109837 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:02.109807 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:02.306826 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:02.306798 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:02.332861 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:02.332819 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wcvxt"] Jul 08 03:09:02.333216 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:02.333203 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fad0396-3157-4f5e-8f4d-19d383c0308b" containerName="authorino" Jul 08 03:09:02.333263 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:02.333218 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fad0396-3157-4f5e-8f4d-19d383c0308b" containerName="authorino" Jul 08 03:09:02.333303 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:02.333294 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fad0396-3157-4f5e-8f4d-19d383c0308b" containerName="authorino" Jul 08 03:09:02.816285 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:02.816248 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wcvxt"] Jul 08 03:09:02.816285 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:02.816287 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wcvxt"] Jul 08 03:09:02.816285 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:02.816300 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6567b85b66-bvxvx"] Jul 08 03:09:02.816593 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:02.816393 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wcvxt" Jul 08 03:09:02.829209 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:02.829182 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6567b85b66-bvxvx"] Jul 08 03:09:02.829353 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:02.829272 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6567b85b66-bvxvx" Jul 08 03:09:02.832427 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:02.832383 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pptkn\"" Jul 08 03:09:02.832575 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:02.832539 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 08 03:09:02.835129 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:02.835108 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wcvxt" Jul 08 03:09:02.859236 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:02.859199 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c0021567-2829-4132-886c-f81169e14a32-tls-cert\") pod \"authorino-6567b85b66-bvxvx\" (UID: \"c0021567-2829-4132-886c-f81169e14a32\") " pod="kuadrant-system/authorino-6567b85b66-bvxvx" Jul 08 03:09:02.859494 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:02.859276 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9xgw\" (UniqueName: \"kubernetes.io/projected/c0021567-2829-4132-886c-f81169e14a32-kube-api-access-n9xgw\") pod \"authorino-6567b85b66-bvxvx\" (UID: \"c0021567-2829-4132-886c-f81169e14a32\") " pod="kuadrant-system/authorino-6567b85b66-bvxvx" Jul 08 03:09:02.959984 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:02.959940 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c0021567-2829-4132-886c-f81169e14a32-tls-cert\") pod \"authorino-6567b85b66-bvxvx\" (UID: \"c0021567-2829-4132-886c-f81169e14a32\") " pod="kuadrant-system/authorino-6567b85b66-bvxvx" Jul 08 03:09:02.960153 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:02.960018 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9xgw\" (UniqueName: \"kubernetes.io/projected/c0021567-2829-4132-886c-f81169e14a32-kube-api-access-n9xgw\") pod \"authorino-6567b85b66-bvxvx\" (UID: \"c0021567-2829-4132-886c-f81169e14a32\") " pod="kuadrant-system/authorino-6567b85b66-bvxvx" Jul 08 03:09:02.962453 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:02.962424 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c0021567-2829-4132-886c-f81169e14a32-tls-cert\") pod \"authorino-6567b85b66-bvxvx\" (UID: \"c0021567-2829-4132-886c-f81169e14a32\") " pod="kuadrant-system/authorino-6567b85b66-bvxvx" Jul 08 03:09:02.970333 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:02.970312 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9xgw\" (UniqueName: \"kubernetes.io/projected/c0021567-2829-4132-886c-f81169e14a32-kube-api-access-n9xgw\") pod \"authorino-6567b85b66-bvxvx\" (UID: \"c0021567-2829-4132-886c-f81169e14a32\") " pod="kuadrant-system/authorino-6567b85b66-bvxvx" Jul 08 03:09:03.139459 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:03.139426 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6567b85b66-bvxvx" Jul 08 03:09:03.271291 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:03.271265 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6567b85b66-bvxvx"] Jul 08 03:09:03.273764 ip-10-0-128-70 kubenswrapper[2564]: W0708 03:09:03.273734 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0021567_2829_4132_886c_f81169e14a32.slice/crio-943f17fcbfac260a678632ade894edd0aeddb5928febb00d16e66b7770674d6a WatchSource:0}: Error finding container 943f17fcbfac260a678632ade894edd0aeddb5928febb00d16e66b7770674d6a: Status 404 returned error can't find the container with id 943f17fcbfac260a678632ade894edd0aeddb5928febb00d16e66b7770674d6a Jul 08 03:09:03.481984 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:03.481879 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6567b85b66-bvxvx" event={"ID":"c0021567-2829-4132-886c-f81169e14a32","Type":"ContainerStarted","Data":"943f17fcbfac260a678632ade894edd0aeddb5928febb00d16e66b7770674d6a"} Jul 08 03:09:03.481984 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:03.481896 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wcvxt" Jul 08 03:09:03.512123 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:03.512090 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wcvxt"] Jul 08 03:09:03.518365 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:03.518336 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wcvxt"] Jul 08 03:09:04.488675 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:04.488642 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6567b85b66-bvxvx" event={"ID":"c0021567-2829-4132-886c-f81169e14a32","Type":"ContainerStarted","Data":"2a5b10617ee99ba3e9a95131fa8f718fa915c3c07fe24efdd2e6a2916b5b5b16"} Jul 08 03:09:04.507285 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:04.507234 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6567b85b66-bvxvx" podStartSLOduration=2.142668972 podStartE2EDuration="2.507216065s" totalImagesPullingTime="364.547093ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:09:04.506299863 +0000 UTC m=+619.163675734" watchObservedRunningTime="2026-07-08 03:09:04.507216065 +0000 UTC m=+619.164591935" Jul 08 03:09:08.727089 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:08.727015 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:08.825504 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:08.825456 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:08.915673 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:08.915642 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:09.237981 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:09.237946 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:12.034189 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:12.034151 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:12.163601 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:12.163563 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:15.327576 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:15.327545 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:41.631909 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:41.631870 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:42.113408 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:42.113314 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:42.610226 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:42.610192 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:42.721631 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:42.721597 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:42.822781 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:42.822748 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:50.712231 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:50.712199 2564 generic.go:356] "Generic (PLEG): container finished" podID="631c38ff-a249-470e-b1c5-bf4f761e4b71" containerID="7cf581d45abfc4cd24e9be816455af990d628df0e3eb748a273ad95d53ed558d" exitCode=0 Jul 08 03:09:50.712633 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:50.712238 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" event={"ID":"631c38ff-a249-470e-b1c5-bf4f761e4b71","Type":"ContainerDied","Data":"7cf581d45abfc4cd24e9be816455af990d628df0e3eb748a273ad95d53ed558d"} Jul 08 03:09:50.712715 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:50.712632 2564 scope.go:117] "RemoveContainer" containerID="7cf581d45abfc4cd24e9be816455af990d628df0e3eb748a273ad95d53ed558d" Jul 08 03:09:50.713622 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:50.713592 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:09:50.713684 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:50.713665 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:09:50.713718 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:50.713700 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:09:51.719854 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:51.719819 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" event={"ID":"631c38ff-a249-470e-b1c5-bf4f761e4b71","Type":"ContainerStarted","Data":"69528072f4f1edba8bfaa2d34d7170aa09f8e87b7bd5c8d2de35f519d7c2ab41"} Jul 08 03:09:52.275441 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:52.275392 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:09:52.275707 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:52.275676 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-lzlg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Jul 08 03:09:52.275802 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:52.275755 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" podUID="631c38ff-a249-470e-b1c5-bf4f761e4b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Jul 08 03:09:53.001996 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:53.001949 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:53.101817 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:53.101780 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:09:53.276046 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:53.275949 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-lzlg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Jul 08 03:09:53.276305 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:53.276278 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" podUID="631c38ff-a249-470e-b1c5-bf4f761e4b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Jul 08 03:09:54.276460 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:54.276422 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-lzlg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Jul 08 03:09:54.277094 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:54.277062 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" podUID="631c38ff-a249-470e-b1c5-bf4f761e4b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Jul 08 03:09:55.276440 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:55.276396 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-lzlg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body= Jul 08 03:09:55.276748 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:55.276718 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" podUID="631c38ff-a249-470e-b1c5-bf4f761e4b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" Jul 08 03:09:56.279313 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:56.279274 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:09:56.279789 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:56.279665 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:09:56.280470 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:56.280454 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lzlg7" Jul 08 03:09:58.044715 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:09:58.044682 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:10:21.175671 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:21.175614 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:10:28.823710 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:28.823676 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:10:39.222160 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:39.222127 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:10:41.221097 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.221066 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8"] Jul 08 03:10:41.224529 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.224508 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8" Jul 08 03:10:41.228813 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.228789 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 08 03:10:41.228978 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.228818 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 08 03:10:41.228978 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.228788 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mv7gx\"" Jul 08 03:10:41.228978 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.228954 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 08 03:10:41.233859 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.233834 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8"] Jul 08 03:10:41.259671 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.259630 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:10:41.332127 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.332091 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3af5c35f-19c9-4917-84d1-a5c432424f0f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8\" (UID: \"3af5c35f-19c9-4917-84d1-a5c432424f0f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8" Jul 08 03:10:41.332305 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.332142 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv7bg\" (UniqueName: \"kubernetes.io/projected/3af5c35f-19c9-4917-84d1-a5c432424f0f-kube-api-access-xv7bg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8\" (UID: \"3af5c35f-19c9-4917-84d1-a5c432424f0f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8" Jul 08 03:10:41.332305 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.332218 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3af5c35f-19c9-4917-84d1-a5c432424f0f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8\" (UID: \"3af5c35f-19c9-4917-84d1-a5c432424f0f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8" Jul 08 03:10:41.332305 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.332252 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3af5c35f-19c9-4917-84d1-a5c432424f0f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8\" (UID: \"3af5c35f-19c9-4917-84d1-a5c432424f0f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8" Jul 08 03:10:41.332305 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.332284 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3af5c35f-19c9-4917-84d1-a5c432424f0f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8\" (UID: \"3af5c35f-19c9-4917-84d1-a5c432424f0f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8" Jul 08 03:10:41.332456 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.332343 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3af5c35f-19c9-4917-84d1-a5c432424f0f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8\" (UID: \"3af5c35f-19c9-4917-84d1-a5c432424f0f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8" Jul 08 03:10:41.433444 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.433403 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3af5c35f-19c9-4917-84d1-a5c432424f0f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8\" (UID: \"3af5c35f-19c9-4917-84d1-a5c432424f0f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8" Jul 08 03:10:41.433623 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.433467 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3af5c35f-19c9-4917-84d1-a5c432424f0f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8\" (UID: \"3af5c35f-19c9-4917-84d1-a5c432424f0f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8" Jul 08 03:10:41.433623 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.433507 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv7bg\" (UniqueName: \"kubernetes.io/projected/3af5c35f-19c9-4917-84d1-a5c432424f0f-kube-api-access-xv7bg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8\" (UID: \"3af5c35f-19c9-4917-84d1-a5c432424f0f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8" Jul 08 03:10:41.433623 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.433534 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3af5c35f-19c9-4917-84d1-a5c432424f0f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8\" (UID: \"3af5c35f-19c9-4917-84d1-a5c432424f0f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8" Jul 08 03:10:41.433623 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.433559 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3af5c35f-19c9-4917-84d1-a5c432424f0f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8\" (UID: \"3af5c35f-19c9-4917-84d1-a5c432424f0f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8" Jul 08 03:10:41.433623 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.433603 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3af5c35f-19c9-4917-84d1-a5c432424f0f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8\" (UID: \"3af5c35f-19c9-4917-84d1-a5c432424f0f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8" Jul 08 03:10:41.433882 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.433857 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3af5c35f-19c9-4917-84d1-a5c432424f0f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8\" (UID: \"3af5c35f-19c9-4917-84d1-a5c432424f0f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8" Jul 08 03:10:41.433959 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.433908 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3af5c35f-19c9-4917-84d1-a5c432424f0f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8\" (UID: \"3af5c35f-19c9-4917-84d1-a5c432424f0f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8" Jul 08 03:10:41.434046 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.434029 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3af5c35f-19c9-4917-84d1-a5c432424f0f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8\" (UID: \"3af5c35f-19c9-4917-84d1-a5c432424f0f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8" Jul 08 03:10:41.436087 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.436069 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3af5c35f-19c9-4917-84d1-a5c432424f0f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8\" (UID: \"3af5c35f-19c9-4917-84d1-a5c432424f0f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8" Jul 08 03:10:41.436246 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.436228 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3af5c35f-19c9-4917-84d1-a5c432424f0f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8\" (UID: \"3af5c35f-19c9-4917-84d1-a5c432424f0f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8" Jul 08 03:10:41.442409 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.442381 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv7bg\" (UniqueName: \"kubernetes.io/projected/3af5c35f-19c9-4917-84d1-a5c432424f0f-kube-api-access-xv7bg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8\" (UID: \"3af5c35f-19c9-4917-84d1-a5c432424f0f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8" Jul 08 03:10:41.536302 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.536200 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8" Jul 08 03:10:41.673009 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.672983 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8"] Jul 08 03:10:41.675661 ip-10-0-128-70 kubenswrapper[2564]: W0708 03:10:41.675633 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af5c35f_19c9_4917_84d1_a5c432424f0f.slice/crio-2d260889511d494e91973933532ee16c9e7cb46070aca85fbbffd4bbb4cc63fe WatchSource:0}: Error finding container 2d260889511d494e91973933532ee16c9e7cb46070aca85fbbffd4bbb4cc63fe: Status 404 returned error can't find the container with id 2d260889511d494e91973933532ee16c9e7cb46070aca85fbbffd4bbb4cc63fe Jul 08 03:10:41.676831 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.676813 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 03:10:41.945966 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:41.945910 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8" event={"ID":"3af5c35f-19c9-4917-84d1-a5c432424f0f","Type":"ContainerStarted","Data":"2d260889511d494e91973933532ee16c9e7cb46070aca85fbbffd4bbb4cc63fe"} Jul 08 03:10:42.952415 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:42.952353 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8" event={"ID":"3af5c35f-19c9-4917-84d1-a5c432424f0f","Type":"ContainerStarted","Data":"08f05d6d5746be7e76c196cd472ba4f774f04cfe089a61d58d0d4637e7d9b9d6"} Jul 08 03:10:51.536488 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:51.536448 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8" Jul 08 03:10:51.536954 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:51.536524 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8" Jul 08 03:10:51.549221 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:51.549194 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8" Jul 08 03:10:51.568168 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:51.568117 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8" podStartSLOduration=9.490777826 podStartE2EDuration="10.5681027s" totalImagesPullingTime="1.077324874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:10:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:10:42.973266079 +0000 UTC m=+717.630641951" watchObservedRunningTime="2026-07-08 03:10:51.5681027 +0000 UTC m=+726.225478571" Jul 08 03:10:52.004856 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:10:52.004826 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8" Jul 08 03:11:04.426767 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:04.426722 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw"] Jul 08 03:11:04.431245 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:04.431218 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw" Jul 08 03:11:04.433721 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:04.433697 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 08 03:11:04.438044 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:04.438016 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw"] Jul 08 03:11:04.463609 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:04.463579 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:11:04.544297 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:04.544251 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36f04153-1335-43f6-b684-f30e05c3e76c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2q7sw\" (UID: \"36f04153-1335-43f6-b684-f30e05c3e76c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw" Jul 08 03:11:04.544297 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:04.544295 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36f04153-1335-43f6-b684-f30e05c3e76c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2q7sw\" (UID: \"36f04153-1335-43f6-b684-f30e05c3e76c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw" Jul 08 03:11:04.544531 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:04.544323 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36f04153-1335-43f6-b684-f30e05c3e76c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2q7sw\" (UID: \"36f04153-1335-43f6-b684-f30e05c3e76c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw" Jul 08 03:11:04.544531 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:04.544400 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36f04153-1335-43f6-b684-f30e05c3e76c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2q7sw\" (UID: \"36f04153-1335-43f6-b684-f30e05c3e76c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw" Jul 08 03:11:04.544531 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:04.544476 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36f04153-1335-43f6-b684-f30e05c3e76c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2q7sw\" (UID: \"36f04153-1335-43f6-b684-f30e05c3e76c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw" Jul 08 03:11:04.544531 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:04.544516 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2qq6\" (UniqueName: \"kubernetes.io/projected/36f04153-1335-43f6-b684-f30e05c3e76c-kube-api-access-p2qq6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2q7sw\" (UID: \"36f04153-1335-43f6-b684-f30e05c3e76c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw" Jul 08 03:11:04.645604 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:04.645570 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36f04153-1335-43f6-b684-f30e05c3e76c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2q7sw\" (UID: \"36f04153-1335-43f6-b684-f30e05c3e76c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw" Jul 08 03:11:04.645791 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:04.645629 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36f04153-1335-43f6-b684-f30e05c3e76c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2q7sw\" (UID: \"36f04153-1335-43f6-b684-f30e05c3e76c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw" Jul 08 03:11:04.645846 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:04.645780 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2qq6\" (UniqueName: \"kubernetes.io/projected/36f04153-1335-43f6-b684-f30e05c3e76c-kube-api-access-p2qq6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2q7sw\" (UID: \"36f04153-1335-43f6-b684-f30e05c3e76c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw" Jul 08 03:11:04.645906 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:04.645888 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36f04153-1335-43f6-b684-f30e05c3e76c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2q7sw\" (UID: \"36f04153-1335-43f6-b684-f30e05c3e76c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw" Jul 08 03:11:04.645999 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:04.645950 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36f04153-1335-43f6-b684-f30e05c3e76c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2q7sw\" (UID: \"36f04153-1335-43f6-b684-f30e05c3e76c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw" Jul 08 03:11:04.646058 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:04.645996 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36f04153-1335-43f6-b684-f30e05c3e76c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2q7sw\" (UID: \"36f04153-1335-43f6-b684-f30e05c3e76c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw" Jul 08 03:11:04.646151 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:04.646132 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36f04153-1335-43f6-b684-f30e05c3e76c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2q7sw\" (UID: \"36f04153-1335-43f6-b684-f30e05c3e76c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw" Jul 08 03:11:04.646327 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:04.646307 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36f04153-1335-43f6-b684-f30e05c3e76c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2q7sw\" (UID: \"36f04153-1335-43f6-b684-f30e05c3e76c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw" Jul 08 03:11:04.646327 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:04.646321 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36f04153-1335-43f6-b684-f30e05c3e76c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2q7sw\" (UID: \"36f04153-1335-43f6-b684-f30e05c3e76c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw" Jul 08 03:11:04.648063 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:04.648044 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36f04153-1335-43f6-b684-f30e05c3e76c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2q7sw\" (UID: \"36f04153-1335-43f6-b684-f30e05c3e76c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw" Jul 08 03:11:04.648584 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:04.648566 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36f04153-1335-43f6-b684-f30e05c3e76c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2q7sw\" (UID: \"36f04153-1335-43f6-b684-f30e05c3e76c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw" Jul 08 03:11:04.654661 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:04.654635 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2qq6\" (UniqueName: \"kubernetes.io/projected/36f04153-1335-43f6-b684-f30e05c3e76c-kube-api-access-p2qq6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2q7sw\" (UID: \"36f04153-1335-43f6-b684-f30e05c3e76c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw" Jul 08 03:11:04.743853 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:04.743765 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw" Jul 08 03:11:04.879494 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:04.879452 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw"] Jul 08 03:11:04.881835 ip-10-0-128-70 kubenswrapper[2564]: W0708 03:11:04.881809 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f04153_1335_43f6_b684_f30e05c3e76c.slice/crio-c2440402bc2ebbdeff1a9d93669ef2b63b19607ba4e37b3d6a59528c03a9046f WatchSource:0}: Error finding container c2440402bc2ebbdeff1a9d93669ef2b63b19607ba4e37b3d6a59528c03a9046f: Status 404 returned error can't find the container with id c2440402bc2ebbdeff1a9d93669ef2b63b19607ba4e37b3d6a59528c03a9046f Jul 08 03:11:05.052433 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:05.052359 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw" event={"ID":"36f04153-1335-43f6-b684-f30e05c3e76c","Type":"ContainerStarted","Data":"c2440402bc2ebbdeff1a9d93669ef2b63b19607ba4e37b3d6a59528c03a9046f"} Jul 08 03:11:06.059104 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:06.059062 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw" event={"ID":"36f04153-1335-43f6-b684-f30e05c3e76c","Type":"ContainerStarted","Data":"9445e67e73677cea02ba6b0bd74eed0162bd094d7e3bc5b0f281e93c902461a2"} Jul 08 03:11:06.077183 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:06.077131 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw" podStartSLOduration=1.882727788 podStartE2EDuration="2.0771145s" totalImagesPullingTime="194.386712ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:11:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:11:06.075268342 +0000 UTC m=+740.732644214" watchObservedRunningTime="2026-07-08 03:11:06.0771145 +0000 UTC m=+740.734490370" Jul 08 03:11:14.744085 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:14.744039 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw" Jul 08 03:11:14.744547 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:14.744107 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw" Jul 08 03:11:14.756505 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:14.756476 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw" Jul 08 03:11:15.116022 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:15.115942 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2q7sw" Jul 08 03:11:17.543647 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:17.543610 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz"] Jul 08 03:11:17.546947 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:17.546914 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz" Jul 08 03:11:17.549988 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:17.549957 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 08 03:11:17.566254 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:17.566223 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e35a160f-a467-4c57-b6e3-ec79232b3b23-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz\" (UID: \"e35a160f-a467-4c57-b6e3-ec79232b3b23\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz" Jul 08 03:11:17.566412 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:17.566267 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e35a160f-a467-4c57-b6e3-ec79232b3b23-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz\" (UID: \"e35a160f-a467-4c57-b6e3-ec79232b3b23\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz" Jul 08 03:11:17.566412 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:17.566320 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e35a160f-a467-4c57-b6e3-ec79232b3b23-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz\" (UID: \"e35a160f-a467-4c57-b6e3-ec79232b3b23\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz" Jul 08 03:11:17.566412 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:17.566361 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vwq5\" (UniqueName: \"kubernetes.io/projected/e35a160f-a467-4c57-b6e3-ec79232b3b23-kube-api-access-4vwq5\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz\" (UID: \"e35a160f-a467-4c57-b6e3-ec79232b3b23\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz" Jul 08 03:11:17.566520 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:17.566421 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e35a160f-a467-4c57-b6e3-ec79232b3b23-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz\" (UID: \"e35a160f-a467-4c57-b6e3-ec79232b3b23\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz" Jul 08 03:11:17.566520 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:17.566480 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e35a160f-a467-4c57-b6e3-ec79232b3b23-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz\" (UID: \"e35a160f-a467-4c57-b6e3-ec79232b3b23\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz" Jul 08 03:11:17.573106 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:17.573071 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz"] Jul 08 03:11:17.644595 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:17.644556 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:11:17.667697 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:17.667635 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e35a160f-a467-4c57-b6e3-ec79232b3b23-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz\" (UID: \"e35a160f-a467-4c57-b6e3-ec79232b3b23\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz" Jul 08 03:11:17.667890 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:17.667735 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vwq5\" (UniqueName: \"kubernetes.io/projected/e35a160f-a467-4c57-b6e3-ec79232b3b23-kube-api-access-4vwq5\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz\" (UID: \"e35a160f-a467-4c57-b6e3-ec79232b3b23\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz" Jul 08 03:11:17.667890 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:17.667771 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e35a160f-a467-4c57-b6e3-ec79232b3b23-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz\" (UID: \"e35a160f-a467-4c57-b6e3-ec79232b3b23\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz" Jul 08 03:11:17.667890 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:17.667821 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e35a160f-a467-4c57-b6e3-ec79232b3b23-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz\" (UID: \"e35a160f-a467-4c57-b6e3-ec79232b3b23\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz" Jul 08 03:11:17.667890 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:17.667851 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e35a160f-a467-4c57-b6e3-ec79232b3b23-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz\" (UID: \"e35a160f-a467-4c57-b6e3-ec79232b3b23\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz" Jul 08 03:11:17.668166 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:17.667897 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e35a160f-a467-4c57-b6e3-ec79232b3b23-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz\" (UID: \"e35a160f-a467-4c57-b6e3-ec79232b3b23\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz" Jul 08 03:11:17.668260 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:17.668236 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e35a160f-a467-4c57-b6e3-ec79232b3b23-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz\" (UID: \"e35a160f-a467-4c57-b6e3-ec79232b3b23\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz" Jul 08 03:11:17.668316 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:17.668278 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e35a160f-a467-4c57-b6e3-ec79232b3b23-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz\" (UID: \"e35a160f-a467-4c57-b6e3-ec79232b3b23\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz" Jul 08 03:11:17.668370 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:17.668331 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e35a160f-a467-4c57-b6e3-ec79232b3b23-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz\" (UID: \"e35a160f-a467-4c57-b6e3-ec79232b3b23\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz" Jul 08 03:11:17.670213 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:17.670193 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e35a160f-a467-4c57-b6e3-ec79232b3b23-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz\" (UID: \"e35a160f-a467-4c57-b6e3-ec79232b3b23\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz" Jul 08 03:11:17.670595 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:17.670575 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e35a160f-a467-4c57-b6e3-ec79232b3b23-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz\" (UID: \"e35a160f-a467-4c57-b6e3-ec79232b3b23\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz" Jul 08 03:11:17.676576 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:17.676549 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vwq5\" (UniqueName: \"kubernetes.io/projected/e35a160f-a467-4c57-b6e3-ec79232b3b23-kube-api-access-4vwq5\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz\" (UID: \"e35a160f-a467-4c57-b6e3-ec79232b3b23\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz" Jul 08 03:11:17.856996 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:17.856874 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz" Jul 08 03:11:17.995607 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:17.995578 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz"] Jul 08 03:11:17.997274 ip-10-0-128-70 kubenswrapper[2564]: W0708 03:11:17.997244 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode35a160f_a467_4c57_b6e3_ec79232b3b23.slice/crio-8917acc4984a618988c7bdf878c81c367a22b2a3840948b98c88a93bc2cb6ae6 WatchSource:0}: Error finding container 8917acc4984a618988c7bdf878c81c367a22b2a3840948b98c88a93bc2cb6ae6: Status 404 returned error can't find the container with id 8917acc4984a618988c7bdf878c81c367a22b2a3840948b98c88a93bc2cb6ae6 Jul 08 03:11:18.119946 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:18.119832 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz" event={"ID":"e35a160f-a467-4c57-b6e3-ec79232b3b23","Type":"ContainerStarted","Data":"8917acc4984a618988c7bdf878c81c367a22b2a3840948b98c88a93bc2cb6ae6"} Jul 08 03:11:19.131971 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:19.131915 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz" event={"ID":"e35a160f-a467-4c57-b6e3-ec79232b3b23","Type":"ContainerStarted","Data":"24e4eec3c06724c6b1b186c1e38a766e26d7e20232ac3bcd5cfc00c609cd2960"} Jul 08 03:11:19.151210 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:19.151155 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz" podStartSLOduration=1.977980553 podStartE2EDuration="2.151136959s" totalImagesPullingTime="173.156406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:11:19.149099282 +0000 UTC m=+753.806475166" watchObservedRunningTime="2026-07-08 03:11:19.151136959 +0000 UTC m=+753.808512830" Jul 08 03:11:27.857430 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:27.857387 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz" Jul 08 03:11:27.857430 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:27.857443 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz" Jul 08 03:11:27.874560 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:27.874534 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz" Jul 08 03:11:28.186454 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:28.186419 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz" Jul 08 03:11:45.264454 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:45.264422 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:11:45.321033 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:45.320983 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:11:47.070294 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:11:47.070265 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:12:14.779681 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:12:14.779586 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6567b85b66-bvxvx"] Jul 08 03:12:14.780379 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:12:14.779769 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-6567b85b66-bvxvx" podUID="c0021567-2829-4132-886c-f81169e14a32" containerName="authorino" containerID="cri-o://2a5b10617ee99ba3e9a95131fa8f718fa915c3c07fe24efdd2e6a2916b5b5b16" gracePeriod=30 Jul 08 03:12:15.025480 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:12:15.025454 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6567b85b66-bvxvx" Jul 08 03:12:15.090606 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:12:15.090450 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c0021567-2829-4132-886c-f81169e14a32-tls-cert\") pod \"c0021567-2829-4132-886c-f81169e14a32\" (UID: \"c0021567-2829-4132-886c-f81169e14a32\") " Jul 08 03:12:15.090751 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:12:15.090626 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9xgw\" (UniqueName: \"kubernetes.io/projected/c0021567-2829-4132-886c-f81169e14a32-kube-api-access-n9xgw\") pod \"c0021567-2829-4132-886c-f81169e14a32\" (UID: \"c0021567-2829-4132-886c-f81169e14a32\") " Jul 08 03:12:15.092711 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:12:15.092679 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0021567-2829-4132-886c-f81169e14a32-kube-api-access-n9xgw" (OuterVolumeSpecName: "kube-api-access-n9xgw") pod "c0021567-2829-4132-886c-f81169e14a32" (UID: "c0021567-2829-4132-886c-f81169e14a32"). InnerVolumeSpecName "kube-api-access-n9xgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:12:15.100526 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:12:15.100501 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0021567-2829-4132-886c-f81169e14a32-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "c0021567-2829-4132-886c-f81169e14a32" (UID: "c0021567-2829-4132-886c-f81169e14a32"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:12:15.192263 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:12:15.192223 2564 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c0021567-2829-4132-886c-f81169e14a32-tls-cert\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:12:15.192263 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:12:15.192254 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9xgw\" (UniqueName: \"kubernetes.io/projected/c0021567-2829-4132-886c-f81169e14a32-kube-api-access-n9xgw\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:12:15.266791 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:12:15.266747 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:12:15.361529 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:12:15.361443 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:12:15.401683 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:12:15.401652 2564 generic.go:356] "Generic (PLEG): container finished" podID="c0021567-2829-4132-886c-f81169e14a32" containerID="2a5b10617ee99ba3e9a95131fa8f718fa915c3c07fe24efdd2e6a2916b5b5b16" exitCode=0 Jul 08 03:12:15.401866 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:12:15.401702 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6567b85b66-bvxvx" Jul 08 03:12:15.401866 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:12:15.401717 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6567b85b66-bvxvx" event={"ID":"c0021567-2829-4132-886c-f81169e14a32","Type":"ContainerDied","Data":"2a5b10617ee99ba3e9a95131fa8f718fa915c3c07fe24efdd2e6a2916b5b5b16"} Jul 08 03:12:15.401866 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:12:15.401749 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6567b85b66-bvxvx" event={"ID":"c0021567-2829-4132-886c-f81169e14a32","Type":"ContainerDied","Data":"943f17fcbfac260a678632ade894edd0aeddb5928febb00d16e66b7770674d6a"} Jul 08 03:12:15.401866 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:12:15.401767 2564 scope.go:117] "RemoveContainer" containerID="2a5b10617ee99ba3e9a95131fa8f718fa915c3c07fe24efdd2e6a2916b5b5b16" Jul 08 03:12:15.411102 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:12:15.411083 2564 scope.go:117] "RemoveContainer" containerID="2a5b10617ee99ba3e9a95131fa8f718fa915c3c07fe24efdd2e6a2916b5b5b16" Jul 08 03:12:15.411392 ip-10-0-128-70 kubenswrapper[2564]: E0708 03:12:15.411370 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a5b10617ee99ba3e9a95131fa8f718fa915c3c07fe24efdd2e6a2916b5b5b16\": container with ID starting with 2a5b10617ee99ba3e9a95131fa8f718fa915c3c07fe24efdd2e6a2916b5b5b16 not found: ID does not exist" containerID="2a5b10617ee99ba3e9a95131fa8f718fa915c3c07fe24efdd2e6a2916b5b5b16" Jul 08 03:12:15.411465 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:12:15.411401 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a5b10617ee99ba3e9a95131fa8f718fa915c3c07fe24efdd2e6a2916b5b5b16"} err="failed to get container status \"2a5b10617ee99ba3e9a95131fa8f718fa915c3c07fe24efdd2e6a2916b5b5b16\": rpc error: code = NotFound desc = could not find container \"2a5b10617ee99ba3e9a95131fa8f718fa915c3c07fe24efdd2e6a2916b5b5b16\": container with ID starting with 2a5b10617ee99ba3e9a95131fa8f718fa915c3c07fe24efdd2e6a2916b5b5b16 not found: ID does not exist" Jul 08 03:12:15.425072 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:12:15.425042 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6567b85b66-bvxvx"] Jul 08 03:12:15.428305 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:12:15.428278 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6567b85b66-bvxvx"] Jul 08 03:12:15.886134 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:12:15.886102 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0021567-2829-4132-886c-f81169e14a32" path="/var/lib/kubelet/pods/c0021567-2829-4132-886c-f81169e14a32/volumes" Jul 08 03:12:16.439907 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:12:16.439865 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:12:16.597862 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:12:16.597827 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:12:24.613766 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:12:24.613728 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:12:24.715335 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:12:24.715294 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:26.173577 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:13:26.173540 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:27.240276 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:13:27.240240 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:27.340897 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:13:27.340849 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:27.763315 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:13:27.763280 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:13:53.436623 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:13:53.436581 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:14:08.273380 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:14:08.273337 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:14:08.371484 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:14:08.371453 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:14:08.517311 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:14:08.517282 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:14:14.249658 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:14:14.249622 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:14:14.355229 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:14:14.355188 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:14:14.461355 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:14:14.461323 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:14:14.657567 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:14:14.657536 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:14:20.301797 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:14:20.301759 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:14:20.409273 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:14:20.409239 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:16:49.295888 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:16:49.295851 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:16:50.792281 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:16:50.792235 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:44.397737 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:18:44.397695 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:18:54.369672 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:18:54.369618 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:21:50.893014 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:21:50.892980 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:21:52.324230 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:21:52.324188 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:22:00.611510 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:22:00.611467 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:00.031135 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:00.031102 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:04.560107 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:04.560059 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:12.807972 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:12.807899 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:17.992079 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:17.992024 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:20.673804 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:20.673769 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:20.773521 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:20.773463 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:21.120521 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:21.120442 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:26.923726 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:26.923690 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:26.981582 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:26.981550 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:27.101366 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.101330 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm"] Jul 08 03:25:27.101962 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.101911 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0021567-2829-4132-886c-f81169e14a32" containerName="authorino" Jul 08 03:25:27.102044 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.101966 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0021567-2829-4132-886c-f81169e14a32" containerName="authorino" Jul 08 03:25:27.102100 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.102062 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0021567-2829-4132-886c-f81169e14a32" containerName="authorino" Jul 08 03:25:27.105807 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.105782 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.108681 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.108652 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-261cc1da-openshift-default-dockercfg-bfmvn\"" Jul 08 03:25:27.128976 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.128948 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm"] Jul 08 03:25:27.150887 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.150857 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:27.238704 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.238616 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:27.280254 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.280209 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51f96b89-c82f-4b29-a189-dbed459bd24e-istio-token\") pod \"e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.280441 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.280261 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-workload-socket\") pod \"e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.280441 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.280350 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-workload-certs\") pod \"e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.280441 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.280405 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-istio-envoy\") pod \"e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.280441 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.280436 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjsgp\" (UniqueName: \"kubernetes.io/projected/51f96b89-c82f-4b29-a189-dbed459bd24e-kube-api-access-vjsgp\") pod \"e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.280663 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.280464 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-istio-data\") pod \"e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.280663 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.280482 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51f96b89-c82f-4b29-a189-dbed459bd24e-istiod-ca-cert\") pod \"e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.280663 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.280522 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-credential-socket\") pod \"e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.280663 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.280541 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51f96b89-c82f-4b29-a189-dbed459bd24e-istio-podinfo\") pod \"e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.381259 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.381216 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51f96b89-c82f-4b29-a189-dbed459bd24e-istio-token\") pod \"e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.381259 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.381261 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-workload-socket\") pod \"e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.381498 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.381291 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-workload-certs\") pod \"e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.381498 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.381335 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-istio-envoy\") pod \"e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.381498 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.381356 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjsgp\" (UniqueName: \"kubernetes.io/projected/51f96b89-c82f-4b29-a189-dbed459bd24e-kube-api-access-vjsgp\") pod \"e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.381498 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.381388 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-istio-data\") pod \"e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.381693 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.381561 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51f96b89-c82f-4b29-a189-dbed459bd24e-istiod-ca-cert\") pod \"e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.381693 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.381637 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-credential-socket\") pod \"e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.381693 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.381663 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51f96b89-c82f-4b29-a189-dbed459bd24e-istio-podinfo\") pod \"e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.381830 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.381696 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-workload-socket\") pod \"e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.381830 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.381743 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-workload-certs\") pod \"e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.381830 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.381797 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-istio-data\") pod \"e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.381996 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.381974 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-credential-socket\") pod \"e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.382332 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.382308 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51f96b89-c82f-4b29-a189-dbed459bd24e-istiod-ca-cert\") pod \"e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.383819 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.383804 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-istio-envoy\") pod \"e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.384477 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.384457 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51f96b89-c82f-4b29-a189-dbed459bd24e-istio-podinfo\") pod \"e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.389881 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.389862 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51f96b89-c82f-4b29-a189-dbed459bd24e-istio-token\") pod \"e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.390310 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.390290 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjsgp\" (UniqueName: \"kubernetes.io/projected/51f96b89-c82f-4b29-a189-dbed459bd24e-kube-api-access-vjsgp\") pod \"e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.417209 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.417169 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:27.561880 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.561852 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm"] Jul 08 03:25:27.564107 ip-10-0-128-70 kubenswrapper[2564]: W0708 03:25:27.564072 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51f96b89_c82f_4b29_a189_dbed459bd24e.slice/crio-fce2d63c8ea093655e223f5ed874cf055b90f03a823003b135ac1dab0eafd9d9 WatchSource:0}: Error finding container fce2d63c8ea093655e223f5ed874cf055b90f03a823003b135ac1dab0eafd9d9: Status 404 returned error can't find the container with id fce2d63c8ea093655e223f5ed874cf055b90f03a823003b135ac1dab0eafd9d9 Jul 08 03:25:27.566236 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.566201 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:25:27.566332 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.566269 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:25:27.566332 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:27.566296 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:25:28.080849 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:28.080795 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" event={"ID":"51f96b89-c82f-4b29-a189-dbed459bd24e","Type":"ContainerStarted","Data":"50a0ed7de7c8cdd960360e7ca2d634516047dc7e55a4127a343c50d0c7ed662c"} Jul 08 03:25:28.081304 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:28.080876 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" event={"ID":"51f96b89-c82f-4b29-a189-dbed459bd24e","Type":"ContainerStarted","Data":"fce2d63c8ea093655e223f5ed874cf055b90f03a823003b135ac1dab0eafd9d9"} Jul 08 03:25:28.103196 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:28.103141 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" podStartSLOduration=1.103121274 podStartE2EDuration="1.103121274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:25:28.101992468 +0000 UTC m=+1602.759368339" watchObservedRunningTime="2026-07-08 03:25:28.103121274 +0000 UTC m=+1602.760497144" Jul 08 03:25:28.254394 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:28.254356 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:28.354942 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:28.354817 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:28.418146 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:28.418110 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:28.422946 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:28.422897 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:28.538667 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:28.538630 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:28.978134 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:28.978096 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:29.077739 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:29.077701 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:29.086214 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:29.086183 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:29.087131 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:29.087115 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:30.693422 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:30.693389 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:36.335247 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:36.335206 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:36.436817 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:36.436733 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:36.512954 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:36.512903 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm"] Jul 08 03:25:36.513159 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:36.513135 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" podUID="51f96b89-c82f-4b29-a189-dbed459bd24e" containerName="istio-proxy" containerID="cri-o://50a0ed7de7c8cdd960360e7ca2d634516047dc7e55a4127a343c50d0c7ed662c" gracePeriod=30 Jul 08 03:25:36.578678 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:36.578627 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:36.683827 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:36.683787 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:38.216543 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:38.216501 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:40.358034 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:40.358010 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:40.417307 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:40.417218 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-istio-data\") pod \"51f96b89-c82f-4b29-a189-dbed459bd24e\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " Jul 08 03:25:40.417307 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:40.417261 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51f96b89-c82f-4b29-a189-dbed459bd24e-istio-podinfo\") pod \"51f96b89-c82f-4b29-a189-dbed459bd24e\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " Jul 08 03:25:40.417307 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:40.417292 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-credential-socket\") pod \"51f96b89-c82f-4b29-a189-dbed459bd24e\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " Jul 08 03:25:40.417669 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:40.417331 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-istio-envoy\") pod \"51f96b89-c82f-4b29-a189-dbed459bd24e\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " Jul 08 03:25:40.417669 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:40.417348 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51f96b89-c82f-4b29-a189-dbed459bd24e-istiod-ca-cert\") pod \"51f96b89-c82f-4b29-a189-dbed459bd24e\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " Jul 08 03:25:40.417669 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:40.417372 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-workload-certs\") pod \"51f96b89-c82f-4b29-a189-dbed459bd24e\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " Jul 08 03:25:40.417669 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:40.417399 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51f96b89-c82f-4b29-a189-dbed459bd24e-istio-token\") pod \"51f96b89-c82f-4b29-a189-dbed459bd24e\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " Jul 08 03:25:40.417669 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:40.417421 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-workload-socket\") pod \"51f96b89-c82f-4b29-a189-dbed459bd24e\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " Jul 08 03:25:40.417669 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:40.417457 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjsgp\" (UniqueName: \"kubernetes.io/projected/51f96b89-c82f-4b29-a189-dbed459bd24e-kube-api-access-vjsgp\") pod \"51f96b89-c82f-4b29-a189-dbed459bd24e\" (UID: \"51f96b89-c82f-4b29-a189-dbed459bd24e\") " Jul 08 03:25:40.417669 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:40.417622 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "51f96b89-c82f-4b29-a189-dbed459bd24e" (UID: "51f96b89-c82f-4b29-a189-dbed459bd24e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:25:40.418058 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:40.417797 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-credential-socket\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:25:40.418058 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:40.417836 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51f96b89-c82f-4b29-a189-dbed459bd24e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "51f96b89-c82f-4b29-a189-dbed459bd24e" (UID: "51f96b89-c82f-4b29-a189-dbed459bd24e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:25:40.418058 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:40.418013 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "51f96b89-c82f-4b29-a189-dbed459bd24e" (UID: "51f96b89-c82f-4b29-a189-dbed459bd24e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:25:40.418383 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:40.418079 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "51f96b89-c82f-4b29-a189-dbed459bd24e" (UID: "51f96b89-c82f-4b29-a189-dbed459bd24e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:25:40.418383 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:40.418159 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-istio-data" (OuterVolumeSpecName: "istio-data") pod "51f96b89-c82f-4b29-a189-dbed459bd24e" (UID: "51f96b89-c82f-4b29-a189-dbed459bd24e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:25:40.420077 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:40.420040 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "51f96b89-c82f-4b29-a189-dbed459bd24e" (UID: "51f96b89-c82f-4b29-a189-dbed459bd24e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:25:40.420077 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:40.420060 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f96b89-c82f-4b29-a189-dbed459bd24e-kube-api-access-vjsgp" (OuterVolumeSpecName: "kube-api-access-vjsgp") pod "51f96b89-c82f-4b29-a189-dbed459bd24e" (UID: "51f96b89-c82f-4b29-a189-dbed459bd24e"). InnerVolumeSpecName "kube-api-access-vjsgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:25:40.420228 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:40.420079 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f96b89-c82f-4b29-a189-dbed459bd24e-istio-token" (OuterVolumeSpecName: "istio-token") pod "51f96b89-c82f-4b29-a189-dbed459bd24e" (UID: "51f96b89-c82f-4b29-a189-dbed459bd24e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:25:40.420228 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:40.420133 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51f96b89-c82f-4b29-a189-dbed459bd24e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "51f96b89-c82f-4b29-a189-dbed459bd24e" (UID: "51f96b89-c82f-4b29-a189-dbed459bd24e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 03:25:40.518967 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:40.518900 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-istio-data\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:25:40.518967 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:40.518960 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51f96b89-c82f-4b29-a189-dbed459bd24e-istio-podinfo\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:25:40.518967 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:40.518969 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-istio-envoy\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:25:40.518967 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:40.518977 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51f96b89-c82f-4b29-a189-dbed459bd24e-istiod-ca-cert\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:25:40.519240 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:40.518985 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-workload-certs\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:25:40.519240 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:40.518993 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51f96b89-c82f-4b29-a189-dbed459bd24e-istio-token\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:25:40.519240 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:40.519001 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51f96b89-c82f-4b29-a189-dbed459bd24e-workload-socket\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:25:40.519240 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:40.519009 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjsgp\" (UniqueName: \"kubernetes.io/projected/51f96b89-c82f-4b29-a189-dbed459bd24e-kube-api-access-vjsgp\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:25:41.144016 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:41.143980 2564 generic.go:356] "Generic (PLEG): container finished" podID="51f96b89-c82f-4b29-a189-dbed459bd24e" containerID="50a0ed7de7c8cdd960360e7ca2d634516047dc7e55a4127a343c50d0c7ed662c" exitCode=0 Jul 08 03:25:41.144196 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:41.144025 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" Jul 08 03:25:41.144196 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:41.144073 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" event={"ID":"51f96b89-c82f-4b29-a189-dbed459bd24e","Type":"ContainerDied","Data":"50a0ed7de7c8cdd960360e7ca2d634516047dc7e55a4127a343c50d0c7ed662c"} Jul 08 03:25:41.144196 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:41.144126 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm" event={"ID":"51f96b89-c82f-4b29-a189-dbed459bd24e","Type":"ContainerDied","Data":"fce2d63c8ea093655e223f5ed874cf055b90f03a823003b135ac1dab0eafd9d9"} Jul 08 03:25:41.144196 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:41.144148 2564 scope.go:117] "RemoveContainer" containerID="50a0ed7de7c8cdd960360e7ca2d634516047dc7e55a4127a343c50d0c7ed662c" Jul 08 03:25:41.153414 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:41.153393 2564 scope.go:117] "RemoveContainer" containerID="50a0ed7de7c8cdd960360e7ca2d634516047dc7e55a4127a343c50d0c7ed662c" Jul 08 03:25:41.153706 ip-10-0-128-70 kubenswrapper[2564]: E0708 03:25:41.153686 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50a0ed7de7c8cdd960360e7ca2d634516047dc7e55a4127a343c50d0c7ed662c\": container with ID starting with 50a0ed7de7c8cdd960360e7ca2d634516047dc7e55a4127a343c50d0c7ed662c not found: ID does not exist" containerID="50a0ed7de7c8cdd960360e7ca2d634516047dc7e55a4127a343c50d0c7ed662c" Jul 08 03:25:41.153762 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:41.153713 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50a0ed7de7c8cdd960360e7ca2d634516047dc7e55a4127a343c50d0c7ed662c"} err="failed to get container status \"50a0ed7de7c8cdd960360e7ca2d634516047dc7e55a4127a343c50d0c7ed662c\": rpc error: code = NotFound desc = could not find container \"50a0ed7de7c8cdd960360e7ca2d634516047dc7e55a4127a343c50d0c7ed662c\": container with ID starting with 50a0ed7de7c8cdd960360e7ca2d634516047dc7e55a4127a343c50d0c7ed662c not found: ID does not exist" Jul 08 03:25:41.175091 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:41.175021 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm"] Jul 08 03:25:41.177300 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:41.177275 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-261cc1da-openshift-default-7c74946fcc-5j8rm"] Jul 08 03:25:41.885996 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:41.885966 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51f96b89-c82f-4b29-a189-dbed459bd24e" path="/var/lib/kubelet/pods/51f96b89-c82f-4b29-a189-dbed459bd24e/volumes" Jul 08 03:25:42.386470 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.386398 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:42.491488 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.491455 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:42.557788 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.557756 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg"] Jul 08 03:25:42.558171 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.558155 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51f96b89-c82f-4b29-a189-dbed459bd24e" containerName="istio-proxy" Jul 08 03:25:42.558171 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.558172 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="51f96b89-c82f-4b29-a189-dbed459bd24e" containerName="istio-proxy" Jul 08 03:25:42.558315 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.558256 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="51f96b89-c82f-4b29-a189-dbed459bd24e" containerName="istio-proxy" Jul 08 03:25:42.561315 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.561294 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:42.564158 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.564132 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-17a37341-openshift-default-dockercfg-klgwl\"" Jul 08 03:25:42.582077 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.579628 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg"] Jul 08 03:25:42.615514 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.615478 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:42.640114 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.640008 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-credential-socket\") pod \"e2e-ait-17a37341-openshift-default-64f7577697-ghbbg\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:42.640114 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.640071 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-workload-certs\") pod \"e2e-ait-17a37341-openshift-default-64f7577697-ghbbg\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:42.640312 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.640148 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istio-envoy\") pod \"e2e-ait-17a37341-openshift-default-64f7577697-ghbbg\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:42.640312 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.640207 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istio-podinfo\") pod \"e2e-ait-17a37341-openshift-default-64f7577697-ghbbg\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:42.640312 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.640272 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istio-token\") pod \"e2e-ait-17a37341-openshift-default-64f7577697-ghbbg\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:42.640312 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.640298 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzfdd\" (UniqueName: \"kubernetes.io/projected/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-kube-api-access-kzfdd\") pod \"e2e-ait-17a37341-openshift-default-64f7577697-ghbbg\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:42.640441 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.640325 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-workload-socket\") pod \"e2e-ait-17a37341-openshift-default-64f7577697-ghbbg\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:42.640441 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.640342 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istiod-ca-cert\") pod \"e2e-ait-17a37341-openshift-default-64f7577697-ghbbg\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:42.640441 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.640362 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istio-data\") pod \"e2e-ait-17a37341-openshift-default-64f7577697-ghbbg\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:42.686587 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.686548 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:42.740847 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.740814 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-credential-socket\") pod \"e2e-ait-17a37341-openshift-default-64f7577697-ghbbg\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:42.741055 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.740864 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-workload-certs\") pod \"e2e-ait-17a37341-openshift-default-64f7577697-ghbbg\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:42.741055 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.740898 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istio-envoy\") pod \"e2e-ait-17a37341-openshift-default-64f7577697-ghbbg\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:42.741055 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.740968 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istio-podinfo\") pod \"e2e-ait-17a37341-openshift-default-64f7577697-ghbbg\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:42.741055 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.740999 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istio-token\") pod \"e2e-ait-17a37341-openshift-default-64f7577697-ghbbg\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:42.741055 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.741018 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzfdd\" (UniqueName: \"kubernetes.io/projected/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-kube-api-access-kzfdd\") pod \"e2e-ait-17a37341-openshift-default-64f7577697-ghbbg\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:42.741055 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.741045 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-workload-socket\") pod \"e2e-ait-17a37341-openshift-default-64f7577697-ghbbg\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:42.741369 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.741068 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istiod-ca-cert\") pod \"e2e-ait-17a37341-openshift-default-64f7577697-ghbbg\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:42.741369 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.741091 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istio-data\") pod \"e2e-ait-17a37341-openshift-default-64f7577697-ghbbg\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:42.741369 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.741288 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-credential-socket\") pod \"e2e-ait-17a37341-openshift-default-64f7577697-ghbbg\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:42.741532 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.741467 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-workload-socket\") pod \"e2e-ait-17a37341-openshift-default-64f7577697-ghbbg\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:42.741582 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.741558 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istio-data\") pod \"e2e-ait-17a37341-openshift-default-64f7577697-ghbbg\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:42.741667 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.741644 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-workload-certs\") pod \"e2e-ait-17a37341-openshift-default-64f7577697-ghbbg\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:42.742152 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.742128 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istiod-ca-cert\") pod \"e2e-ait-17a37341-openshift-default-64f7577697-ghbbg\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:42.743453 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.743431 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istio-envoy\") pod \"e2e-ait-17a37341-openshift-default-64f7577697-ghbbg\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:42.743660 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.743642 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istio-podinfo\") pod \"e2e-ait-17a37341-openshift-default-64f7577697-ghbbg\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:42.750255 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.750231 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istio-token\") pod \"e2e-ait-17a37341-openshift-default-64f7577697-ghbbg\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:42.750449 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.750425 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzfdd\" (UniqueName: \"kubernetes.io/projected/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-kube-api-access-kzfdd\") pod \"e2e-ait-17a37341-openshift-default-64f7577697-ghbbg\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:42.873468 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:42.873429 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:43.017352 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:43.017324 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg"] Jul 08 03:25:43.019963 ip-10-0-128-70 kubenswrapper[2564]: W0708 03:25:43.019935 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed215a5_bb0e_4675_b1a5_aa0914bb131f.slice/crio-d91cbc56b82c9055b21d505654884b20fe90a9f96ec376314ca05d06024c880e WatchSource:0}: Error finding container d91cbc56b82c9055b21d505654884b20fe90a9f96ec376314ca05d06024c880e: Status 404 returned error can't find the container with id d91cbc56b82c9055b21d505654884b20fe90a9f96ec376314ca05d06024c880e Jul 08 03:25:43.022238 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:43.022202 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:25:43.022332 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:43.022274 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:25:43.022332 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:43.022305 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:25:43.156730 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:43.156693 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" event={"ID":"8ed215a5-bb0e-4675-b1a5-aa0914bb131f","Type":"ContainerStarted","Data":"0220e349c4bb47357869c2783c8b9f12845ff677572eb76ab013a9e729caa1ab"} Jul 08 03:25:43.156960 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:43.156747 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" event={"ID":"8ed215a5-bb0e-4675-b1a5-aa0914bb131f","Type":"ContainerStarted","Data":"d91cbc56b82c9055b21d505654884b20fe90a9f96ec376314ca05d06024c880e"} Jul 08 03:25:43.179245 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:43.179138 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" podStartSLOduration=1.179120055 podStartE2EDuration="1.179120055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:25:43.178193116 +0000 UTC m=+1617.835568987" watchObservedRunningTime="2026-07-08 03:25:43.179120055 +0000 UTC m=+1617.836495926" Jul 08 03:25:43.714498 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:43.714463 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:43.798327 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:43.798273 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:43.874031 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:43.873995 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:43.883407 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:43.883377 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:44.404512 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:44.404475 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:44.505221 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:44.505188 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:44.620820 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:44.620786 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:44.878873 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:44.878831 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:45.168578 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:45.168498 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:45.169576 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:45.169556 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:45.757377 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:45.757343 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:51.807518 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:51.807478 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:51.876836 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:51.876789 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:52.719254 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:52.719206 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg"] Jul 08 03:25:52.719462 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:52.719399 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" podUID="8ed215a5-bb0e-4675-b1a5-aa0914bb131f" containerName="istio-proxy" containerID="cri-o://0220e349c4bb47357869c2783c8b9f12845ff677572eb76ab013a9e729caa1ab" gracePeriod=30 Jul 08 03:25:52.749285 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:52.749250 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:52.850094 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:52.850052 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:54.375550 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:54.375522 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:54.459495 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:54.459456 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istio-podinfo\") pod \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " Jul 08 03:25:54.459683 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:54.459547 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-workload-certs\") pod \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " Jul 08 03:25:54.459683 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:54.459569 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzfdd\" (UniqueName: \"kubernetes.io/projected/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-kube-api-access-kzfdd\") pod \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " Jul 08 03:25:54.459683 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:54.459591 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istio-envoy\") pod \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " Jul 08 03:25:54.459683 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:54.459606 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istiod-ca-cert\") pod \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " Jul 08 03:25:54.459683 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:54.459627 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istio-token\") pod \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " Jul 08 03:25:54.459683 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:54.459667 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-credential-socket\") pod \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " Jul 08 03:25:54.460063 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:54.459692 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istio-data\") pod \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " Jul 08 03:25:54.460063 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:54.459719 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-workload-socket\") pod \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\" (UID: \"8ed215a5-bb0e-4675-b1a5-aa0914bb131f\") " Jul 08 03:25:54.460063 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:54.459813 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8ed215a5-bb0e-4675-b1a5-aa0914bb131f" (UID: "8ed215a5-bb0e-4675-b1a5-aa0914bb131f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:25:54.460228 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:54.460082 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-workload-certs\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:25:54.460228 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:54.460085 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8ed215a5-bb0e-4675-b1a5-aa0914bb131f" (UID: "8ed215a5-bb0e-4675-b1a5-aa0914bb131f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:25:54.460938 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:54.460337 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8ed215a5-bb0e-4675-b1a5-aa0914bb131f" (UID: "8ed215a5-bb0e-4675-b1a5-aa0914bb131f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:25:54.461367 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:54.461339 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istio-data" (OuterVolumeSpecName: "istio-data") pod "8ed215a5-bb0e-4675-b1a5-aa0914bb131f" (UID: "8ed215a5-bb0e-4675-b1a5-aa0914bb131f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:25:54.461692 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:54.461659 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8ed215a5-bb0e-4675-b1a5-aa0914bb131f" (UID: "8ed215a5-bb0e-4675-b1a5-aa0914bb131f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:25:54.462431 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:54.462368 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istio-token" (OuterVolumeSpecName: "istio-token") pod "8ed215a5-bb0e-4675-b1a5-aa0914bb131f" (UID: "8ed215a5-bb0e-4675-b1a5-aa0914bb131f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:25:54.462570 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:54.462545 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8ed215a5-bb0e-4675-b1a5-aa0914bb131f" (UID: "8ed215a5-bb0e-4675-b1a5-aa0914bb131f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:25:54.462642 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:54.462593 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8ed215a5-bb0e-4675-b1a5-aa0914bb131f" (UID: "8ed215a5-bb0e-4675-b1a5-aa0914bb131f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 03:25:54.462642 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:54.462601 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-kube-api-access-kzfdd" (OuterVolumeSpecName: "kube-api-access-kzfdd") pod "8ed215a5-bb0e-4675-b1a5-aa0914bb131f" (UID: "8ed215a5-bb0e-4675-b1a5-aa0914bb131f"). InnerVolumeSpecName "kube-api-access-kzfdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:25:54.560999 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:54.560962 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzfdd\" (UniqueName: \"kubernetes.io/projected/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-kube-api-access-kzfdd\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:25:54.560999 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:54.560995 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istio-envoy\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:25:54.560999 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:54.561005 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istiod-ca-cert\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:25:54.561282 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:54.561013 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istio-token\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:25:54.561282 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:54.561022 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-credential-socket\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:25:54.561282 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:54.561031 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istio-data\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:25:54.561282 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:54.561039 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-workload-socket\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:25:54.561282 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:54.561046 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ed215a5-bb0e-4675-b1a5-aa0914bb131f-istio-podinfo\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:25:55.222227 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:55.222192 2564 generic.go:356] "Generic (PLEG): container finished" podID="8ed215a5-bb0e-4675-b1a5-aa0914bb131f" containerID="0220e349c4bb47357869c2783c8b9f12845ff677572eb76ab013a9e729caa1ab" exitCode=0 Jul 08 03:25:55.222409 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:55.222254 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" Jul 08 03:25:55.222409 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:55.222274 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" event={"ID":"8ed215a5-bb0e-4675-b1a5-aa0914bb131f","Type":"ContainerDied","Data":"0220e349c4bb47357869c2783c8b9f12845ff677572eb76ab013a9e729caa1ab"} Jul 08 03:25:55.222409 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:55.222326 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg" event={"ID":"8ed215a5-bb0e-4675-b1a5-aa0914bb131f","Type":"ContainerDied","Data":"d91cbc56b82c9055b21d505654884b20fe90a9f96ec376314ca05d06024c880e"} Jul 08 03:25:55.222409 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:55.222348 2564 scope.go:117] "RemoveContainer" containerID="0220e349c4bb47357869c2783c8b9f12845ff677572eb76ab013a9e729caa1ab" Jul 08 03:25:55.231299 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:55.231283 2564 scope.go:117] "RemoveContainer" containerID="0220e349c4bb47357869c2783c8b9f12845ff677572eb76ab013a9e729caa1ab" Jul 08 03:25:55.231566 ip-10-0-128-70 kubenswrapper[2564]: E0708 03:25:55.231545 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0220e349c4bb47357869c2783c8b9f12845ff677572eb76ab013a9e729caa1ab\": container with ID starting with 0220e349c4bb47357869c2783c8b9f12845ff677572eb76ab013a9e729caa1ab not found: ID does not exist" containerID="0220e349c4bb47357869c2783c8b9f12845ff677572eb76ab013a9e729caa1ab" Jul 08 03:25:55.231639 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:55.231573 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0220e349c4bb47357869c2783c8b9f12845ff677572eb76ab013a9e729caa1ab"} err="failed to get container status \"0220e349c4bb47357869c2783c8b9f12845ff677572eb76ab013a9e729caa1ab\": rpc error: code = NotFound desc = could not find container \"0220e349c4bb47357869c2783c8b9f12845ff677572eb76ab013a9e729caa1ab\": container with ID starting with 0220e349c4bb47357869c2783c8b9f12845ff677572eb76ab013a9e729caa1ab not found: ID does not exist" Jul 08 03:25:55.244547 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:55.244518 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg"] Jul 08 03:25:55.248055 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:55.248031 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-17a37341-openshift-default-64f7577697-ghbbg"] Jul 08 03:25:55.886878 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:55.886831 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ed215a5-bb0e-4675-b1a5-aa0914bb131f" path="/var/lib/kubelet/pods/8ed215a5-bb0e-4675-b1a5-aa0914bb131f/volumes" Jul 08 03:25:56.744005 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:56.743972 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:58.751905 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:58.751868 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:58.854085 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:58.854049 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:58.934339 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:58.934305 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf"] Jul 08 03:25:58.934872 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:58.934851 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ed215a5-bb0e-4675-b1a5-aa0914bb131f" containerName="istio-proxy" Jul 08 03:25:58.934872 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:58.934872 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ed215a5-bb0e-4675-b1a5-aa0914bb131f" containerName="istio-proxy" Jul 08 03:25:58.935061 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:58.934975 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ed215a5-bb0e-4675-b1a5-aa0914bb131f" containerName="istio-proxy" Jul 08 03:25:58.937970 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:58.937951 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:58.940817 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:58.940795 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-b93aa927-openshift-default-dockercfg-9hrdm\"" Jul 08 03:25:58.950071 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:58.950047 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf"] Jul 08 03:25:58.964569 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:58.964534 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:59.004035 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.003938 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-workload-certs\") pod \"e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:59.004035 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.003998 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-istio-data\") pod \"e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:59.004035 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.004027 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f285b849-f9c6-48a5-8635-dd69bb320d99-istio-token\") pod \"e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:59.004352 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.004061 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f285b849-f9c6-48a5-8635-dd69bb320d99-istiod-ca-cert\") pod \"e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:59.004352 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.004110 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-credential-socket\") pod \"e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:59.004352 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.004139 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-istio-envoy\") pod \"e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:59.004352 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.004244 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc92x\" (UniqueName: \"kubernetes.io/projected/f285b849-f9c6-48a5-8635-dd69bb320d99-kube-api-access-gc92x\") pod \"e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:59.004352 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.004291 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f285b849-f9c6-48a5-8635-dd69bb320d99-istio-podinfo\") pod \"e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:59.004352 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.004325 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-workload-socket\") pod \"e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:59.064138 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.064102 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:25:59.105359 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.105316 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f285b849-f9c6-48a5-8635-dd69bb320d99-istiod-ca-cert\") pod \"e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:59.105359 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.105363 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-credential-socket\") pod \"e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:59.105599 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.105380 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-istio-envoy\") pod \"e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:59.105599 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.105435 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc92x\" (UniqueName: \"kubernetes.io/projected/f285b849-f9c6-48a5-8635-dd69bb320d99-kube-api-access-gc92x\") pod \"e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:59.105599 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.105459 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f285b849-f9c6-48a5-8635-dd69bb320d99-istio-podinfo\") pod \"e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:59.105599 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.105484 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-workload-socket\") pod \"e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:59.105599 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.105565 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-workload-certs\") pod \"e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:59.105599 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.105595 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-istio-data\") pod \"e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:59.105894 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.105627 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f285b849-f9c6-48a5-8635-dd69bb320d99-istio-token\") pod \"e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:59.105894 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.105877 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-credential-socket\") pod \"e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:59.106140 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.106096 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-workload-socket\") pod \"e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:59.106140 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.106104 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-workload-certs\") pod \"e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:59.106445 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.106167 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-istio-data\") pod \"e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:59.106445 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.106255 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f285b849-f9c6-48a5-8635-dd69bb320d99-istiod-ca-cert\") pod \"e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:59.108033 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.108005 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-istio-envoy\") pod \"e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:59.108119 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.108100 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f285b849-f9c6-48a5-8635-dd69bb320d99-istio-podinfo\") pod \"e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:59.114055 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.114031 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f285b849-f9c6-48a5-8635-dd69bb320d99-istio-token\") pod \"e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:59.114299 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.114279 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc92x\" (UniqueName: \"kubernetes.io/projected/f285b849-f9c6-48a5-8635-dd69bb320d99-kube-api-access-gc92x\") pod \"e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:59.250600 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.250566 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:25:59.415380 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.415353 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf"] Jul 08 03:25:59.417466 ip-10-0-128-70 kubenswrapper[2564]: W0708 03:25:59.417441 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf285b849_f9c6_48a5_8635_dd69bb320d99.slice/crio-8f09339b19e470f713a737b8bdc80775e029d860e5e8ae0cf82300aaef058451 WatchSource:0}: Error finding container 8f09339b19e470f713a737b8bdc80775e029d860e5e8ae0cf82300aaef058451: Status 404 returned error can't find the container with id 8f09339b19e470f713a737b8bdc80775e029d860e5e8ae0cf82300aaef058451 Jul 08 03:25:59.419628 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.419594 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:25:59.419719 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.419659 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:25:59.419719 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:25:59.419690 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:26:00.147909 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:00.147877 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:00.244418 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:00.244380 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:00.247300 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:00.247271 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" event={"ID":"f285b849-f9c6-48a5-8635-dd69bb320d99","Type":"ContainerStarted","Data":"3ba94a99f59b92613b227fa82dacac8e2f0954b162f46224f24f722e24bcb18c"} Jul 08 03:26:00.247460 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:00.247327 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" event={"ID":"f285b849-f9c6-48a5-8635-dd69bb320d99","Type":"ContainerStarted","Data":"8f09339b19e470f713a737b8bdc80775e029d860e5e8ae0cf82300aaef058451"} Jul 08 03:26:00.251738 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:00.251719 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:26:00.256760 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:00.256737 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:26:00.273371 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:00.273326 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" podStartSLOduration=2.273310174 podStartE2EDuration="2.273310174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:26:00.271910675 +0000 UTC m=+1634.929286575" watchObservedRunningTime="2026-07-08 03:26:00.273310174 +0000 UTC m=+1634.930686044" Jul 08 03:26:00.316933 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:00.316893 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:00.446228 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:00.446146 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:00.732718 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:00.732631 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:00.850511 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:00.850476 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:01.252242 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:01.252214 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:26:01.253289 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:01.253270 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:26:02.090362 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:02.090331 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:07.513842 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:07.513779 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:07.619978 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:07.619941 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:07.654719 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:07.654686 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf"] Jul 08 03:26:07.654938 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:07.654899 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" podUID="f285b849-f9c6-48a5-8635-dd69bb320d99" containerName="istio-proxy" containerID="cri-o://3ba94a99f59b92613b227fa82dacac8e2f0954b162f46224f24f722e24bcb18c" gracePeriod=30 Jul 08 03:26:07.683143 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:07.683085 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:07.778594 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:07.778432 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:09.317515 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:09.317486 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:11.899530 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:11.899505 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:26:12.041727 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.041642 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-credential-socket\") pod \"f285b849-f9c6-48a5-8635-dd69bb320d99\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " Jul 08 03:26:12.041727 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.041710 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-istio-envoy\") pod \"f285b849-f9c6-48a5-8635-dd69bb320d99\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " Jul 08 03:26:12.041988 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.041737 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-istio-data\") pod \"f285b849-f9c6-48a5-8635-dd69bb320d99\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " Jul 08 03:26:12.041988 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.041810 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f285b849-f9c6-48a5-8635-dd69bb320d99-istiod-ca-cert\") pod \"f285b849-f9c6-48a5-8635-dd69bb320d99\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " Jul 08 03:26:12.041988 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.041841 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f285b849-f9c6-48a5-8635-dd69bb320d99-istio-token\") pod \"f285b849-f9c6-48a5-8635-dd69bb320d99\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " Jul 08 03:26:12.041988 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.041865 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f285b849-f9c6-48a5-8635-dd69bb320d99-istio-podinfo\") pod \"f285b849-f9c6-48a5-8635-dd69bb320d99\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " Jul 08 03:26:12.041988 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.041931 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gc92x\" (UniqueName: \"kubernetes.io/projected/f285b849-f9c6-48a5-8635-dd69bb320d99-kube-api-access-gc92x\") pod \"f285b849-f9c6-48a5-8635-dd69bb320d99\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " Jul 08 03:26:12.042244 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.041979 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f285b849-f9c6-48a5-8635-dd69bb320d99" (UID: "f285b849-f9c6-48a5-8635-dd69bb320d99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:26:12.042244 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.041995 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-workload-certs\") pod \"f285b849-f9c6-48a5-8635-dd69bb320d99\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " Jul 08 03:26:12.042244 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.042037 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-workload-socket\") pod \"f285b849-f9c6-48a5-8635-dd69bb320d99\" (UID: \"f285b849-f9c6-48a5-8635-dd69bb320d99\") " Jul 08 03:26:12.042244 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.042216 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f285b849-f9c6-48a5-8635-dd69bb320d99-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f285b849-f9c6-48a5-8635-dd69bb320d99" (UID: "f285b849-f9c6-48a5-8635-dd69bb320d99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:26:12.042495 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.042352 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f285b849-f9c6-48a5-8635-dd69bb320d99" (UID: "f285b849-f9c6-48a5-8635-dd69bb320d99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:26:12.042495 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.042437 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f285b849-f9c6-48a5-8635-dd69bb320d99-istiod-ca-cert\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:26:12.042495 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.042456 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-workload-certs\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:26:12.042495 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.042471 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-credential-socket\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:26:12.042696 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.042509 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f285b849-f9c6-48a5-8635-dd69bb320d99" (UID: "f285b849-f9c6-48a5-8635-dd69bb320d99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:26:12.042909 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.042878 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-istio-data" (OuterVolumeSpecName: "istio-data") pod "f285b849-f9c6-48a5-8635-dd69bb320d99" (UID: "f285b849-f9c6-48a5-8635-dd69bb320d99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:26:12.044661 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.044632 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f285b849-f9c6-48a5-8635-dd69bb320d99" (UID: "f285b849-f9c6-48a5-8635-dd69bb320d99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:26:12.044788 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.044706 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f285b849-f9c6-48a5-8635-dd69bb320d99-kube-api-access-gc92x" (OuterVolumeSpecName: "kube-api-access-gc92x") pod "f285b849-f9c6-48a5-8635-dd69bb320d99" (UID: "f285b849-f9c6-48a5-8635-dd69bb320d99"). InnerVolumeSpecName "kube-api-access-gc92x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:26:12.044968 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.044945 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f285b849-f9c6-48a5-8635-dd69bb320d99-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f285b849-f9c6-48a5-8635-dd69bb320d99" (UID: "f285b849-f9c6-48a5-8635-dd69bb320d99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 03:26:12.045169 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.045150 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f285b849-f9c6-48a5-8635-dd69bb320d99-istio-token" (OuterVolumeSpecName: "istio-token") pod "f285b849-f9c6-48a5-8635-dd69bb320d99" (UID: "f285b849-f9c6-48a5-8635-dd69bb320d99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:26:12.143517 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.143477 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-istio-envoy\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:26:12.143517 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.143512 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-istio-data\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:26:12.143517 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.143521 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f285b849-f9c6-48a5-8635-dd69bb320d99-istio-token\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:26:12.143517 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.143529 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f285b849-f9c6-48a5-8635-dd69bb320d99-istio-podinfo\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:26:12.143793 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.143539 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc92x\" (UniqueName: \"kubernetes.io/projected/f285b849-f9c6-48a5-8635-dd69bb320d99-kube-api-access-gc92x\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:26:12.143793 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.143549 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f285b849-f9c6-48a5-8635-dd69bb320d99-workload-socket\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:26:12.315515 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.315426 2564 generic.go:356] "Generic (PLEG): container finished" podID="f285b849-f9c6-48a5-8635-dd69bb320d99" containerID="3ba94a99f59b92613b227fa82dacac8e2f0954b162f46224f24f722e24bcb18c" exitCode=0 Jul 08 03:26:12.315515 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.315479 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" Jul 08 03:26:12.315515 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.315485 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" event={"ID":"f285b849-f9c6-48a5-8635-dd69bb320d99","Type":"ContainerDied","Data":"3ba94a99f59b92613b227fa82dacac8e2f0954b162f46224f24f722e24bcb18c"} Jul 08 03:26:12.315759 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.315530 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf" event={"ID":"f285b849-f9c6-48a5-8635-dd69bb320d99","Type":"ContainerDied","Data":"8f09339b19e470f713a737b8bdc80775e029d860e5e8ae0cf82300aaef058451"} Jul 08 03:26:12.315759 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.315550 2564 scope.go:117] "RemoveContainer" containerID="3ba94a99f59b92613b227fa82dacac8e2f0954b162f46224f24f722e24bcb18c" Jul 08 03:26:12.325153 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.325135 2564 scope.go:117] "RemoveContainer" containerID="3ba94a99f59b92613b227fa82dacac8e2f0954b162f46224f24f722e24bcb18c" Jul 08 03:26:12.325425 ip-10-0-128-70 kubenswrapper[2564]: E0708 03:26:12.325407 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ba94a99f59b92613b227fa82dacac8e2f0954b162f46224f24f722e24bcb18c\": container with ID starting with 3ba94a99f59b92613b227fa82dacac8e2f0954b162f46224f24f722e24bcb18c not found: ID does not exist" containerID="3ba94a99f59b92613b227fa82dacac8e2f0954b162f46224f24f722e24bcb18c" Jul 08 03:26:12.325470 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.325434 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ba94a99f59b92613b227fa82dacac8e2f0954b162f46224f24f722e24bcb18c"} err="failed to get container status \"3ba94a99f59b92613b227fa82dacac8e2f0954b162f46224f24f722e24bcb18c\": rpc error: code = NotFound desc = could not find container \"3ba94a99f59b92613b227fa82dacac8e2f0954b162f46224f24f722e24bcb18c\": container with ID starting with 3ba94a99f59b92613b227fa82dacac8e2f0954b162f46224f24f722e24bcb18c not found: ID does not exist" Jul 08 03:26:12.338519 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.338486 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf"] Jul 08 03:26:12.345199 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:12.345168 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-b93aa927-openshift-default-6b6d5f6979-dkvpf"] Jul 08 03:26:13.579962 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:13.579906 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:13.641087 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:13.641055 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:13.886729 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:13.886695 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f285b849-f9c6-48a5-8635-dd69bb320d99" path="/var/lib/kubelet/pods/f285b849-f9c6-48a5-8635-dd69bb320d99/volumes" Jul 08 03:26:14.047818 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:14.047788 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:14.147902 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:14.147823 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:14.250232 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:14.250193 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:45.494634 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:45.494596 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:51.214289 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:51.214253 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:56.692624 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:56.692587 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:26:57.939821 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:26:57.939787 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:27:09.389234 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:27:09.389153 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:27:11.149124 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:27:11.149092 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:27:11.249798 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:27:11.249765 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:27:23.412323 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:27:23.412273 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:27:29.131056 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:27:29.131024 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:27:29.236001 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:27:29.235908 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:27:29.331151 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:27:29.331114 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:27:45.201320 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:27:45.201274 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:27:51.018426 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:27:51.018381 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:27:51.078150 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:27:51.078116 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:27:51.175003 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:27:51.174962 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:27:51.263058 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:27:51.263027 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:28:01.733470 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:28:01.733434 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:28:02.533793 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:28:02.533744 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:28:08.462808 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:28:08.462768 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:28:08.523185 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:28:08.523140 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:28:08.599319 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:28:08.599279 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:28:08.699742 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:28:08.699704 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:28:08.799960 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:28:08.799858 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:28:09.550569 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:28:09.550536 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:28:09.637405 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:28:09.637368 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:28:09.737189 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:28:09.737152 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:28:41.187326 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:28:41.187241 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:28:46.938805 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:28:46.938766 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:28:49.960564 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:28:49.960527 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:28:50.061660 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:28:50.061626 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:28:52.527640 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:28:52.527600 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:28:52.591604 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:28:52.591571 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:28:59.392952 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:28:59.392884 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:28:59.495513 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:28:59.495478 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:29:23.131535 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:23.131496 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:29:28.770711 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:28.770673 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:29:43.483117 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:43.483060 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:29:43.584541 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:43.584499 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:29:43.753636 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:43.753548 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:29:44.036543 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.036464 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2"] Jul 08 03:29:44.036849 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.036836 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f285b849-f9c6-48a5-8635-dd69bb320d99" containerName="istio-proxy" Jul 08 03:29:44.036898 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.036852 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f285b849-f9c6-48a5-8635-dd69bb320d99" containerName="istio-proxy" Jul 08 03:29:44.036959 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.036935 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f285b849-f9c6-48a5-8635-dd69bb320d99" containerName="istio-proxy" Jul 08 03:29:44.039974 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.039957 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.042637 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.042618 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ee0d907f-openshift-default-dockercfg-84lvj\"" Jul 08 03:29:44.061577 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.061551 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2"] Jul 08 03:29:44.064730 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.064702 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:29:44.121168 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.121133 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4cwf\" (UniqueName: \"kubernetes.io/projected/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-kube-api-access-t4cwf\") pod \"e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.121376 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.121248 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-credential-socket\") pod \"e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.121376 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.121306 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-workload-socket\") pod \"e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.121376 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.121334 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istio-envoy\") pod \"e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.121376 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.121373 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istiod-ca-cert\") pod \"e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.121599 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.121484 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-workload-certs\") pod \"e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.121599 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.121513 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istio-token\") pod \"e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.121599 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.121553 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istio-data\") pod \"e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.121599 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.121592 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istio-podinfo\") pod \"e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.222864 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.222815 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-workload-certs\") pod \"e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.223089 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.222873 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istio-token\") pod \"e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.223089 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.222911 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istio-data\") pod \"e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.223089 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.222980 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istio-podinfo\") pod \"e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.223089 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.223030 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4cwf\" (UniqueName: \"kubernetes.io/projected/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-kube-api-access-t4cwf\") pod \"e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.223089 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.223060 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-credential-socket\") pod \"e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.223362 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.223097 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-workload-socket\") pod \"e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.223362 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.223133 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istio-envoy\") pod \"e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.223362 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.223169 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istiod-ca-cert\") pod \"e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.223362 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.223347 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-workload-certs\") pod \"e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.223546 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.223412 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istio-data\") pod \"e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.223546 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.223475 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-credential-socket\") pod \"e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.223546 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.223510 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-workload-socket\") pod \"e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.223909 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.223886 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istiod-ca-cert\") pod \"e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.225502 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.225484 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istio-envoy\") pod \"e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.225634 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.225618 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istio-podinfo\") pod \"e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.231286 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.231262 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istio-token\") pod \"e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.231388 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.231337 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4cwf\" (UniqueName: \"kubernetes.io/projected/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-kube-api-access-t4cwf\") pod \"e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.350756 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.350662 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:44.409031 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.408963 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:29:44.495713 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.495663 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2"] Jul 08 03:29:44.498093 ip-10-0-128-70 kubenswrapper[2564]: W0708 03:29:44.498055 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9086b8ab_a7ba_4a83_8bf4_dd85d3f3e548.slice/crio-48b7ac7645b08a84e39e24afdcf81230f76f4e02421278db2338fa6c1f41d35d WatchSource:0}: Error finding container 48b7ac7645b08a84e39e24afdcf81230f76f4e02421278db2338fa6c1f41d35d: Status 404 returned error can't find the container with id 48b7ac7645b08a84e39e24afdcf81230f76f4e02421278db2338fa6c1f41d35d Jul 08 03:29:44.500183 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.500147 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:29:44.500281 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.500214 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:29:44.500281 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:44.500242 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:29:45.313301 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:45.313269 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" event={"ID":"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548","Type":"ContainerStarted","Data":"391844285c6a87b14c54bcf247e3f1a702f1d1ee00072b7b1c3b61243f2afa2f"} Jul 08 03:29:45.313486 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:45.313317 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" event={"ID":"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548","Type":"ContainerStarted","Data":"48b7ac7645b08a84e39e24afdcf81230f76f4e02421278db2338fa6c1f41d35d"} Jul 08 03:29:45.336129 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:45.336075 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" podStartSLOduration=1.336054302 podStartE2EDuration="1.336054302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:29:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:29:45.333268644 +0000 UTC m=+1859.990644529" watchObservedRunningTime="2026-07-08 03:29:45.336054302 +0000 UTC m=+1859.993430173" Jul 08 03:29:45.351271 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:45.351235 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:45.356066 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:45.356042 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:46.317786 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:46.317755 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:46.318875 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:46.318853 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:29:47.331224 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:47.331189 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:29:47.443093 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:47.443062 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:29:47.524336 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:47.524302 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:29:47.715376 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:47.715334 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:29:48.076547 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:48.076457 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:29:48.183019 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:48.182981 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:29:51.743463 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:51.743425 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:29:52.392839 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:52.392808 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:29:52.493568 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:52.493533 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:29:52.820765 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:52.820670 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:29:58.622294 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:58.622260 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:29:58.739274 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:58.739242 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:29:59.105993 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:59.105954 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:29:59.472945 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:59.472854 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:29:59.566732 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:29:59.566697 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:30:02.359102 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:02.359063 2564 patch_prober.go:28] interesting pod/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 08 03:30:02.363004 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:02.359152 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" podUID="9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 08 03:30:03.408137 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:03.408101 2564 generic.go:356] "Generic (PLEG): container finished" podID="9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548" containerID="391844285c6a87b14c54bcf247e3f1a702f1d1ee00072b7b1c3b61243f2afa2f" exitCode=0 Jul 08 03:30:03.408509 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:03.408176 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" event={"ID":"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548","Type":"ContainerDied","Data":"391844285c6a87b14c54bcf247e3f1a702f1d1ee00072b7b1c3b61243f2afa2f"} Jul 08 03:30:03.408572 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:03.408547 2564 scope.go:117] "RemoveContainer" containerID="391844285c6a87b14c54bcf247e3f1a702f1d1ee00072b7b1c3b61243f2afa2f" Jul 08 03:30:03.409523 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:03.409498 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:30:03.409581 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:03.409566 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:30:03.409616 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:03.409598 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:30:04.414786 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:04.414749 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" event={"ID":"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548","Type":"ContainerStarted","Data":"0486077f90d07f8cd5475bad69145ee634d95353b34848937124c1dc0072aa26"} Jul 08 03:30:05.232854 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:05.232819 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:30:05.294090 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:05.294062 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:30:05.351074 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:05.351035 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:30:05.354399 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:05.354374 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:30:05.419335 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:05.419302 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:30:05.420214 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:05.420196 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:30:05.461680 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:05.461644 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2"] Jul 08 03:30:05.478373 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:05.478339 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:30:05.820828 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:05.820800 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:30:05.921488 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:05.921424 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:30:06.053377 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:06.053314 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:30:06.424879 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:06.424797 2564 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" secret="" err="secret \"e2e-mt-ee0d907f-openshift-default-dockercfg-84lvj\" not found" Jul 08 03:30:07.434274 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:07.434234 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" podUID="9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548" containerName="istio-proxy" containerID="cri-o://0486077f90d07f8cd5475bad69145ee634d95353b34848937124c1dc0072aa26" gracePeriod=30 Jul 08 03:30:11.868248 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:11.868212 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:30:11.976274 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:11.976232 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:30:12.069340 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.069310 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:30:12.214025 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.213995 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:30:12.693965 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.693941 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:30:12.800578 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.800544 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4cwf\" (UniqueName: \"kubernetes.io/projected/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-kube-api-access-t4cwf\") pod \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " Jul 08 03:30:12.800578 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.800581 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-workload-socket\") pod \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " Jul 08 03:30:12.800835 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.800604 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istio-data\") pod \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " Jul 08 03:30:12.800835 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.800637 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istio-podinfo\") pod \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " Jul 08 03:30:12.800835 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.800765 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-workload-certs\") pod \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " Jul 08 03:30:12.800835 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.800830 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istio-token\") pod \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " Jul 08 03:30:12.801067 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.800899 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-credential-socket\") pod \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " Jul 08 03:30:12.801067 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.800936 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548" (UID: "9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:30:12.801067 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.800974 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istiod-ca-cert\") pod \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " Jul 08 03:30:12.801067 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.801011 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istio-envoy\") pod \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\" (UID: \"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548\") " Jul 08 03:30:12.801067 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.801049 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548" (UID: "9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:30:12.801327 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.801138 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548" (UID: "9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:30:12.801529 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.801448 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-workload-certs\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:30:12.801620 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.801540 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-credential-socket\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:30:12.801620 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.801542 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548" (UID: "9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:30:12.801620 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.801555 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-workload-socket\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:30:12.801967 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.801817 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istio-data" (OuterVolumeSpecName: "istio-data") pod "9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548" (UID: "9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:30:12.803163 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.803122 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istio-token" (OuterVolumeSpecName: "istio-token") pod "9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548" (UID: "9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:30:12.803276 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.803189 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548" (UID: "9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 03:30:12.803276 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.803214 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548" (UID: "9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:30:12.803276 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.803225 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-kube-api-access-t4cwf" (OuterVolumeSpecName: "kube-api-access-t4cwf") pod "9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548" (UID: "9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548"). InnerVolumeSpecName "kube-api-access-t4cwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:30:12.902871 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.902799 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4cwf\" (UniqueName: \"kubernetes.io/projected/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-kube-api-access-t4cwf\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:30:12.902871 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.902841 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istio-data\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:30:12.902871 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.902851 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istio-podinfo\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:30:12.902871 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.902858 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istio-token\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:30:12.902871 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.902866 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istiod-ca-cert\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:30:12.902871 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:12.902875 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548-istio-envoy\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:30:13.464002 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:13.463969 2564 generic.go:356] "Generic (PLEG): container finished" podID="9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548" containerID="0486077f90d07f8cd5475bad69145ee634d95353b34848937124c1dc0072aa26" exitCode=0 Jul 08 03:30:13.464154 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:13.464018 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" Jul 08 03:30:13.464154 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:13.464059 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" event={"ID":"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548","Type":"ContainerDied","Data":"0486077f90d07f8cd5475bad69145ee634d95353b34848937124c1dc0072aa26"} Jul 08 03:30:13.464154 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:13.464113 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2" event={"ID":"9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548","Type":"ContainerDied","Data":"48b7ac7645b08a84e39e24afdcf81230f76f4e02421278db2338fa6c1f41d35d"} Jul 08 03:30:13.464154 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:13.464137 2564 scope.go:117] "RemoveContainer" containerID="0486077f90d07f8cd5475bad69145ee634d95353b34848937124c1dc0072aa26" Jul 08 03:30:13.473364 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:13.473345 2564 scope.go:117] "RemoveContainer" containerID="391844285c6a87b14c54bcf247e3f1a702f1d1ee00072b7b1c3b61243f2afa2f" Jul 08 03:30:13.481392 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:13.481370 2564 scope.go:117] "RemoveContainer" containerID="0486077f90d07f8cd5475bad69145ee634d95353b34848937124c1dc0072aa26" Jul 08 03:30:13.481692 ip-10-0-128-70 kubenswrapper[2564]: E0708 03:30:13.481671 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0486077f90d07f8cd5475bad69145ee634d95353b34848937124c1dc0072aa26\": container with ID starting with 0486077f90d07f8cd5475bad69145ee634d95353b34848937124c1dc0072aa26 not found: ID does not exist" containerID="0486077f90d07f8cd5475bad69145ee634d95353b34848937124c1dc0072aa26" Jul 08 03:30:13.481745 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:13.481702 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0486077f90d07f8cd5475bad69145ee634d95353b34848937124c1dc0072aa26"} err="failed to get container status \"0486077f90d07f8cd5475bad69145ee634d95353b34848937124c1dc0072aa26\": rpc error: code = NotFound desc = could not find container \"0486077f90d07f8cd5475bad69145ee634d95353b34848937124c1dc0072aa26\": container with ID starting with 0486077f90d07f8cd5475bad69145ee634d95353b34848937124c1dc0072aa26 not found: ID does not exist" Jul 08 03:30:13.481745 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:13.481721 2564 scope.go:117] "RemoveContainer" containerID="391844285c6a87b14c54bcf247e3f1a702f1d1ee00072b7b1c3b61243f2afa2f" Jul 08 03:30:13.481970 ip-10-0-128-70 kubenswrapper[2564]: E0708 03:30:13.481952 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"391844285c6a87b14c54bcf247e3f1a702f1d1ee00072b7b1c3b61243f2afa2f\": container with ID starting with 391844285c6a87b14c54bcf247e3f1a702f1d1ee00072b7b1c3b61243f2afa2f not found: ID does not exist" containerID="391844285c6a87b14c54bcf247e3f1a702f1d1ee00072b7b1c3b61243f2afa2f" Jul 08 03:30:13.482025 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:13.481978 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"391844285c6a87b14c54bcf247e3f1a702f1d1ee00072b7b1c3b61243f2afa2f"} err="failed to get container status \"391844285c6a87b14c54bcf247e3f1a702f1d1ee00072b7b1c3b61243f2afa2f\": rpc error: code = NotFound desc = could not find container \"391844285c6a87b14c54bcf247e3f1a702f1d1ee00072b7b1c3b61243f2afa2f\": container with ID starting with 391844285c6a87b14c54bcf247e3f1a702f1d1ee00072b7b1c3b61243f2afa2f not found: ID does not exist" Jul 08 03:30:13.486082 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:13.486053 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2"] Jul 08 03:30:13.496148 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:13.492746 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ee0d907f-openshift-default-789f774d8c-zrkm2"] Jul 08 03:30:13.886333 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:13.886301 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548" path="/var/lib/kubelet/pods/9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548/volumes" Jul 08 03:30:17.905613 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:17.905574 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:30:18.004738 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:18.004705 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:30:24.633687 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:24.633647 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:30:24.917800 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:24.917711 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh"] Jul 08 03:30:24.918279 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:24.918263 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548" containerName="istio-proxy" Jul 08 03:30:24.918346 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:24.918283 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548" containerName="istio-proxy" Jul 08 03:30:24.918346 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:24.918305 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548" containerName="istio-proxy" Jul 08 03:30:24.918346 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:24.918314 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548" containerName="istio-proxy" Jul 08 03:30:24.918441 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:24.918407 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548" containerName="istio-proxy" Jul 08 03:30:24.918441 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:24.918424 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9086b8ab-a7ba-4a83-8bf4-dd85d3f3e548" containerName="istio-proxy" Jul 08 03:30:24.921671 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:24.921653 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:24.924271 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:24.924253 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1cf90851-openshift-default-dockercfg-9gp9v\"" Jul 08 03:30:24.932089 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:24.932064 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh"] Jul 08 03:30:24.940564 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:24.940539 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:30:25.002716 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.002679 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:30:25.009015 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.008986 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0bc0aa1-312e-422a-a521-a750541c3f10-istiod-ca-cert\") pod \"e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:25.009094 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.009022 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-istio-envoy\") pod \"e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:25.009094 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.009042 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-workload-socket\") pod \"e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:25.009170 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.009131 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-credential-socket\") pod \"e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:25.009170 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.009160 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-istio-data\") pod \"e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:25.009234 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.009183 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-workload-certs\") pod \"e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:25.009234 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.009202 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0bc0aa1-312e-422a-a521-a750541c3f10-istio-podinfo\") pod \"e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:25.009299 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.009232 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0bc0aa1-312e-422a-a521-a750541c3f10-istio-token\") pod \"e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:25.009299 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.009279 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x82dd\" (UniqueName: \"kubernetes.io/projected/e0bc0aa1-312e-422a-a521-a750541c3f10-kube-api-access-x82dd\") pod \"e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:25.110113 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.110076 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x82dd\" (UniqueName: \"kubernetes.io/projected/e0bc0aa1-312e-422a-a521-a750541c3f10-kube-api-access-x82dd\") pod \"e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:25.110311 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.110157 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0bc0aa1-312e-422a-a521-a750541c3f10-istiod-ca-cert\") pod \"e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:25.110311 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.110184 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-istio-envoy\") pod \"e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:25.110311 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.110206 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-workload-socket\") pod \"e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:25.110451 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.110363 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-credential-socket\") pod \"e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:25.110451 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.110394 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-istio-data\") pod \"e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:25.110451 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.110428 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-workload-certs\") pod \"e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:25.110594 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.110460 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0bc0aa1-312e-422a-a521-a750541c3f10-istio-podinfo\") pod \"e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:25.110594 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.110493 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0bc0aa1-312e-422a-a521-a750541c3f10-istio-token\") pod \"e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:25.110692 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.110639 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-workload-socket\") pod \"e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:25.110692 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.110665 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-credential-socket\") pod \"e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:25.111055 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.111024 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0bc0aa1-312e-422a-a521-a750541c3f10-istiod-ca-cert\") pod \"e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:25.111237 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.111185 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-workload-certs\") pod \"e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:25.111423 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.111354 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-istio-data\") pod \"e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:25.112755 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.112734 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-istio-envoy\") pod \"e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:25.113099 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.113079 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0bc0aa1-312e-422a-a521-a750541c3f10-istio-podinfo\") pod \"e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:25.119259 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.119233 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0bc0aa1-312e-422a-a521-a750541c3f10-istio-token\") pod \"e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:25.119462 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.119438 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x82dd\" (UniqueName: \"kubernetes.io/projected/e0bc0aa1-312e-422a-a521-a750541c3f10-kube-api-access-x82dd\") pod \"e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:25.232978 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.232867 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:25.372813 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.372785 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh"] Jul 08 03:30:25.375336 ip-10-0-128-70 kubenswrapper[2564]: W0708 03:30:25.375309 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0bc0aa1_312e_422a_a521_a750541c3f10.slice/crio-502393c562f79061974442d4b4f2180cb99dae06ca1fb0c2d01ab8e8c1489b6a WatchSource:0}: Error finding container 502393c562f79061974442d4b4f2180cb99dae06ca1fb0c2d01ab8e8c1489b6a: Status 404 returned error can't find the container with id 502393c562f79061974442d4b4f2180cb99dae06ca1fb0c2d01ab8e8c1489b6a Jul 08 03:30:25.377372 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.377337 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:30:25.377470 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.377423 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:30:25.377532 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.377468 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:30:25.522815 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.522720 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" event={"ID":"e0bc0aa1-312e-422a-a521-a750541c3f10","Type":"ContainerStarted","Data":"1175528ac563f3a68ea10b4ae39e048ea9ef1832de937302b2bcfcf69544d6d1"} Jul 08 03:30:25.522815 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.522790 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" event={"ID":"e0bc0aa1-312e-422a-a521-a750541c3f10","Type":"ContainerStarted","Data":"502393c562f79061974442d4b4f2180cb99dae06ca1fb0c2d01ab8e8c1489b6a"} Jul 08 03:30:25.542909 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:25.542853 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" podStartSLOduration=1.5428357400000001 podStartE2EDuration="1.54283574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:30:25.540641079 +0000 UTC m=+1900.198016955" watchObservedRunningTime="2026-07-08 03:30:25.54283574 +0000 UTC m=+1900.200211589" Jul 08 03:30:26.233318 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:26.233281 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:26.238369 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:26.238344 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:26.528154 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:26.528060 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:26.529203 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:26.529185 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:27.506185 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:27.506150 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:30:27.584504 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:27.584469 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:30:27.658144 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:27.658104 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:30:27.814663 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:27.814563 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:30:28.164453 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:28.164401 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:30:28.268517 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:28.268481 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:30:31.897459 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:31.897420 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:30:39.594287 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:39.594255 2564 generic.go:356] "Generic (PLEG): container finished" podID="e0bc0aa1-312e-422a-a521-a750541c3f10" containerID="1175528ac563f3a68ea10b4ae39e048ea9ef1832de937302b2bcfcf69544d6d1" exitCode=0 Jul 08 03:30:39.594716 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:39.594330 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" event={"ID":"e0bc0aa1-312e-422a-a521-a750541c3f10","Type":"ContainerDied","Data":"1175528ac563f3a68ea10b4ae39e048ea9ef1832de937302b2bcfcf69544d6d1"} Jul 08 03:30:39.594716 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:39.594710 2564 scope.go:117] "RemoveContainer" containerID="1175528ac563f3a68ea10b4ae39e048ea9ef1832de937302b2bcfcf69544d6d1" Jul 08 03:30:39.595741 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:39.595714 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:30:39.595802 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:39.595781 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:30:39.595853 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:39.595810 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:30:40.603664 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:40.603625 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" event={"ID":"e0bc0aa1-312e-422a-a521-a750541c3f10","Type":"ContainerStarted","Data":"5be5b2be41da608d0b39e2bfb6d2d86585de182370981b7f3f58ba241873cd7b"} Jul 08 03:30:41.233749 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:41.233709 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:41.234976 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:41.234951 2564 patch_prober.go:28] interesting pod/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 08 03:30:41.235082 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:41.235020 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" podUID="e0bc0aa1-312e-422a-a521-a750541c3f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 08 03:30:41.528137 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:41.528036 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:42.233725 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:42.233687 2564 patch_prober.go:28] interesting pod/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 08 03:30:42.234166 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:42.233749 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" podUID="e0bc0aa1-312e-422a-a521-a750541c3f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 08 03:30:43.233648 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:43.233603 2564 patch_prober.go:28] interesting pod/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 08 03:30:43.233849 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:43.233664 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" podUID="e0bc0aa1-312e-422a-a521-a750541c3f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 08 03:30:44.234060 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:44.234020 2564 patch_prober.go:28] interesting pod/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 08 03:30:44.234559 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:44.234088 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" podUID="e0bc0aa1-312e-422a-a521-a750541c3f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 08 03:30:45.238017 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:45.237983 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:45.238955 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:45.238936 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:30:53.225102 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:53.225060 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv"] Jul 08 03:30:53.228580 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:53.228560 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" Jul 08 03:30:53.232897 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:53.232865 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1cf90851\"/\"openshift-service-ca.crt\"" Jul 08 03:30:53.233128 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:53.232947 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1cf90851\"/\"default-dockercfg-56h7w\"" Jul 08 03:30:53.233128 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:53.232962 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1cf90851\"/\"kube-root-ca.crt\"" Jul 08 03:30:53.233128 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:53.233051 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1cf90851\"/\"e2e-lifecycle-model-1cf90851-kserve-self-signed-certs\"" Jul 08 03:30:53.237708 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:53.237686 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv"] Jul 08 03:30:53.257404 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:53.257358 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:30:53.392214 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:53.392173 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d77382-a8c8-47a9-a0cb-8248a6607338-tmp-dir\") pod \"e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv\" (UID: \"43d77382-a8c8-47a9-a0cb-8248a6607338\") " pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" Jul 08 03:30:53.392548 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:53.392519 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43d77382-a8c8-47a9-a0cb-8248a6607338-home\") pod \"e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv\" (UID: \"43d77382-a8c8-47a9-a0cb-8248a6607338\") " pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" Jul 08 03:30:53.392733 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:53.392710 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9m89\" (UniqueName: \"kubernetes.io/projected/43d77382-a8c8-47a9-a0cb-8248a6607338-kube-api-access-z9m89\") pod \"e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv\" (UID: \"43d77382-a8c8-47a9-a0cb-8248a6607338\") " pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" Jul 08 03:30:53.392911 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:53.392828 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43d77382-a8c8-47a9-a0cb-8248a6607338-tls-certs\") pod \"e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv\" (UID: \"43d77382-a8c8-47a9-a0cb-8248a6607338\") " pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" Jul 08 03:30:53.392911 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:53.392857 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43d77382-a8c8-47a9-a0cb-8248a6607338-dshm\") pod \"e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv\" (UID: \"43d77382-a8c8-47a9-a0cb-8248a6607338\") " pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" Jul 08 03:30:53.393050 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:53.392966 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43d77382-a8c8-47a9-a0cb-8248a6607338-model-cache\") pod \"e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv\" (UID: \"43d77382-a8c8-47a9-a0cb-8248a6607338\") " pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" Jul 08 03:30:53.493833 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:53.493716 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9m89\" (UniqueName: \"kubernetes.io/projected/43d77382-a8c8-47a9-a0cb-8248a6607338-kube-api-access-z9m89\") pod \"e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv\" (UID: \"43d77382-a8c8-47a9-a0cb-8248a6607338\") " pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" Jul 08 03:30:53.493833 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:53.493809 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43d77382-a8c8-47a9-a0cb-8248a6607338-tls-certs\") pod \"e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv\" (UID: \"43d77382-a8c8-47a9-a0cb-8248a6607338\") " pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" Jul 08 03:30:53.493833 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:53.493839 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43d77382-a8c8-47a9-a0cb-8248a6607338-dshm\") pod \"e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv\" (UID: \"43d77382-a8c8-47a9-a0cb-8248a6607338\") " pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" Jul 08 03:30:53.494182 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:53.493886 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43d77382-a8c8-47a9-a0cb-8248a6607338-model-cache\") pod \"e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv\" (UID: \"43d77382-a8c8-47a9-a0cb-8248a6607338\") " pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" Jul 08 03:30:53.494182 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:53.493955 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d77382-a8c8-47a9-a0cb-8248a6607338-tmp-dir\") pod \"e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv\" (UID: \"43d77382-a8c8-47a9-a0cb-8248a6607338\") " pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" Jul 08 03:30:53.494182 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:53.493992 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43d77382-a8c8-47a9-a0cb-8248a6607338-home\") pod \"e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv\" (UID: \"43d77382-a8c8-47a9-a0cb-8248a6607338\") " pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" Jul 08 03:30:53.494384 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:53.494362 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d77382-a8c8-47a9-a0cb-8248a6607338-tmp-dir\") pod \"e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv\" (UID: \"43d77382-a8c8-47a9-a0cb-8248a6607338\") " pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" Jul 08 03:30:53.494447 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:53.494411 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43d77382-a8c8-47a9-a0cb-8248a6607338-model-cache\") pod \"e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv\" (UID: \"43d77382-a8c8-47a9-a0cb-8248a6607338\") " pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" Jul 08 03:30:53.494499 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:53.494445 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43d77382-a8c8-47a9-a0cb-8248a6607338-home\") pod \"e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv\" (UID: \"43d77382-a8c8-47a9-a0cb-8248a6607338\") " pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" Jul 08 03:30:53.496235 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:53.496218 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43d77382-a8c8-47a9-a0cb-8248a6607338-dshm\") pod \"e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv\" (UID: \"43d77382-a8c8-47a9-a0cb-8248a6607338\") " pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" Jul 08 03:30:53.496485 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:53.496469 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43d77382-a8c8-47a9-a0cb-8248a6607338-tls-certs\") pod \"e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv\" (UID: \"43d77382-a8c8-47a9-a0cb-8248a6607338\") " pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" Jul 08 03:30:53.502994 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:53.502975 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9m89\" (UniqueName: \"kubernetes.io/projected/43d77382-a8c8-47a9-a0cb-8248a6607338-kube-api-access-z9m89\") pod \"e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv\" (UID: \"43d77382-a8c8-47a9-a0cb-8248a6607338\") " pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" Jul 08 03:30:53.539555 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:53.539511 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" Jul 08 03:30:53.686878 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:53.686852 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv"] Jul 08 03:30:53.689082 ip-10-0-128-70 kubenswrapper[2564]: W0708 03:30:53.689050 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43d77382_a8c8_47a9_a0cb_8248a6607338.slice/crio-f9091004ce8f24ab474a4ec1781ae6bf4e0e337a329a9ab3ded5e2594c4317bf WatchSource:0}: Error finding container f9091004ce8f24ab474a4ec1781ae6bf4e0e337a329a9ab3ded5e2594c4317bf: Status 404 returned error can't find the container with id f9091004ce8f24ab474a4ec1781ae6bf4e0e337a329a9ab3ded5e2594c4317bf Jul 08 03:30:54.671171 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:54.671130 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" event={"ID":"43d77382-a8c8-47a9-a0cb-8248a6607338","Type":"ContainerStarted","Data":"692a24950466eaae74f3f5257e00d7faa20524f7ad128b68aed52d2399bc938d"} Jul 08 03:30:54.671171 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:54.671177 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" event={"ID":"43d77382-a8c8-47a9-a0cb-8248a6607338","Type":"ContainerStarted","Data":"f9091004ce8f24ab474a4ec1781ae6bf4e0e337a329a9ab3ded5e2594c4317bf"} Jul 08 03:30:54.690204 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:30:54.690144 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" podStartSLOduration=1.690127431 podStartE2EDuration="1.690127431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:30:54.688180765 +0000 UTC m=+1929.345556637" watchObservedRunningTime="2026-07-08 03:30:54.690127431 +0000 UTC m=+1929.347503301" Jul 08 03:31:03.540637 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:03.540592 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" Jul 08 03:31:03.540637 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:03.540641 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" Jul 08 03:31:03.553361 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:03.553332 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" Jul 08 03:31:03.728731 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:03.728699 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" Jul 08 03:31:25.679074 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:25.679039 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:31:26.034628 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:26.034541 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:31:31.725745 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:31.725705 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:31:36.702537 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:36.702446 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv"] Jul 08 03:31:36.799939 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:36.799883 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:31:51.724421 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:51.724357 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" podUID="43d77382-a8c8-47a9-a0cb-8248a6607338" containerName="main" containerID="cri-o://692a24950466eaae74f3f5257e00d7faa20524f7ad128b68aed52d2399bc938d" gracePeriod=45 Jul 08 03:31:51.944283 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:51.944246 2564 generic.go:356] "Generic (PLEG): container finished" podID="43d77382-a8c8-47a9-a0cb-8248a6607338" containerID="692a24950466eaae74f3f5257e00d7faa20524f7ad128b68aed52d2399bc938d" exitCode=0 Jul 08 03:31:51.944454 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:51.944298 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" event={"ID":"43d77382-a8c8-47a9-a0cb-8248a6607338","Type":"ContainerDied","Data":"692a24950466eaae74f3f5257e00d7faa20524f7ad128b68aed52d2399bc938d"} Jul 08 03:31:51.978097 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:51.978030 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" Jul 08 03:31:52.084440 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:52.084406 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43d77382-a8c8-47a9-a0cb-8248a6607338-dshm\") pod \"43d77382-a8c8-47a9-a0cb-8248a6607338\" (UID: \"43d77382-a8c8-47a9-a0cb-8248a6607338\") " Jul 08 03:31:52.084614 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:52.084460 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43d77382-a8c8-47a9-a0cb-8248a6607338-model-cache\") pod \"43d77382-a8c8-47a9-a0cb-8248a6607338\" (UID: \"43d77382-a8c8-47a9-a0cb-8248a6607338\") " Jul 08 03:31:52.084614 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:52.084525 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43d77382-a8c8-47a9-a0cb-8248a6607338-home\") pod \"43d77382-a8c8-47a9-a0cb-8248a6607338\" (UID: \"43d77382-a8c8-47a9-a0cb-8248a6607338\") " Jul 08 03:31:52.084614 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:52.084551 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43d77382-a8c8-47a9-a0cb-8248a6607338-tls-certs\") pod \"43d77382-a8c8-47a9-a0cb-8248a6607338\" (UID: \"43d77382-a8c8-47a9-a0cb-8248a6607338\") " Jul 08 03:31:52.084614 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:52.084567 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d77382-a8c8-47a9-a0cb-8248a6607338-tmp-dir\") pod \"43d77382-a8c8-47a9-a0cb-8248a6607338\" (UID: \"43d77382-a8c8-47a9-a0cb-8248a6607338\") " Jul 08 03:31:52.084831 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:52.084622 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9m89\" (UniqueName: \"kubernetes.io/projected/43d77382-a8c8-47a9-a0cb-8248a6607338-kube-api-access-z9m89\") pod \"43d77382-a8c8-47a9-a0cb-8248a6607338\" (UID: \"43d77382-a8c8-47a9-a0cb-8248a6607338\") " Jul 08 03:31:52.084831 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:52.084729 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d77382-a8c8-47a9-a0cb-8248a6607338-model-cache" (OuterVolumeSpecName: "model-cache") pod "43d77382-a8c8-47a9-a0cb-8248a6607338" (UID: "43d77382-a8c8-47a9-a0cb-8248a6607338"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:31:52.084966 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:52.084844 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d77382-a8c8-47a9-a0cb-8248a6607338-home" (OuterVolumeSpecName: "home") pod "43d77382-a8c8-47a9-a0cb-8248a6607338" (UID: "43d77382-a8c8-47a9-a0cb-8248a6607338"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:31:52.085034 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:52.084998 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43d77382-a8c8-47a9-a0cb-8248a6607338-home\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:31:52.085034 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:52.085017 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43d77382-a8c8-47a9-a0cb-8248a6607338-model-cache\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:31:52.085176 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:52.085151 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d77382-a8c8-47a9-a0cb-8248a6607338-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "43d77382-a8c8-47a9-a0cb-8248a6607338" (UID: "43d77382-a8c8-47a9-a0cb-8248a6607338"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:31:52.086853 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:52.086809 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d77382-a8c8-47a9-a0cb-8248a6607338-dshm" (OuterVolumeSpecName: "dshm") pod "43d77382-a8c8-47a9-a0cb-8248a6607338" (UID: "43d77382-a8c8-47a9-a0cb-8248a6607338"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:31:52.086853 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:52.086827 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43d77382-a8c8-47a9-a0cb-8248a6607338-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43d77382-a8c8-47a9-a0cb-8248a6607338" (UID: "43d77382-a8c8-47a9-a0cb-8248a6607338"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:31:52.087058 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:52.086954 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d77382-a8c8-47a9-a0cb-8248a6607338-kube-api-access-z9m89" (OuterVolumeSpecName: "kube-api-access-z9m89") pod "43d77382-a8c8-47a9-a0cb-8248a6607338" (UID: "43d77382-a8c8-47a9-a0cb-8248a6607338"). InnerVolumeSpecName "kube-api-access-z9m89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:31:52.186094 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:52.186057 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43d77382-a8c8-47a9-a0cb-8248a6607338-dshm\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:31:52.186094 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:52.186088 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43d77382-a8c8-47a9-a0cb-8248a6607338-tls-certs\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:31:52.186297 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:52.186119 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d77382-a8c8-47a9-a0cb-8248a6607338-tmp-dir\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:31:52.186297 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:52.186127 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9m89\" (UniqueName: \"kubernetes.io/projected/43d77382-a8c8-47a9-a0cb-8248a6607338-kube-api-access-z9m89\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:31:52.950839 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:52.950797 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" Jul 08 03:31:52.951322 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:52.950801 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv" event={"ID":"43d77382-a8c8-47a9-a0cb-8248a6607338","Type":"ContainerDied","Data":"f9091004ce8f24ab474a4ec1781ae6bf4e0e337a329a9ab3ded5e2594c4317bf"} Jul 08 03:31:52.951322 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:52.950964 2564 scope.go:117] "RemoveContainer" containerID="692a24950466eaae74f3f5257e00d7faa20524f7ad128b68aed52d2399bc938d" Jul 08 03:31:52.976911 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:52.976877 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv"] Jul 08 03:31:52.979821 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:52.979795 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-1cf90851/e2e-lifecycle-model-1cf90851-kserve-9d86b84c5-vmfxv"] Jul 08 03:31:53.886330 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:31:53.886287 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43d77382-a8c8-47a9-a0cb-8248a6607338" path="/var/lib/kubelet/pods/43d77382-a8c8-47a9-a0cb-8248a6607338/volumes" Jul 08 03:32:12.845718 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:12.845677 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:32:14.158105 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:14.158071 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:32:19.228889 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:19.228856 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:32:24.783589 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:24.783558 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:32:24.851838 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:24.851797 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:32:25.069198 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:25.062802 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh"] Jul 08 03:32:25.069198 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:25.063194 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" podUID="e0bc0aa1-312e-422a-a521-a750541c3f10" containerName="istio-proxy" containerID="cri-o://5be5b2be41da608d0b39e2bfb6d2d86585de182370981b7f3f58ba241873cd7b" gracePeriod=30 Jul 08 03:32:25.086780 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:25.086749 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:32:25.872708 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:25.872669 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:32:26.351353 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:26.351307 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:32:26.446708 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:26.446671 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:32:26.546445 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:26.546410 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:32:27.246912 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:27.246877 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:32:27.345372 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:27.345334 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:32:27.463291 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:27.463244 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:32:30.240376 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.240340 2564 patch_prober.go:28] interesting pod/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 08 03:32:30.240743 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.240430 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" podUID="e0bc0aa1-312e-422a-a521-a750541c3f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 08 03:32:30.330204 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.330171 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:32:30.450374 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.450335 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-workload-certs\") pod \"e0bc0aa1-312e-422a-a521-a750541c3f10\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " Jul 08 03:32:30.450374 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.450377 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0bc0aa1-312e-422a-a521-a750541c3f10-istio-token\") pod \"e0bc0aa1-312e-422a-a521-a750541c3f10\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " Jul 08 03:32:30.450613 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.450434 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-credential-socket\") pod \"e0bc0aa1-312e-422a-a521-a750541c3f10\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " Jul 08 03:32:30.450613 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.450474 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0bc0aa1-312e-422a-a521-a750541c3f10-istio-podinfo\") pod \"e0bc0aa1-312e-422a-a521-a750541c3f10\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " Jul 08 03:32:30.450613 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.450535 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-istio-envoy\") pod \"e0bc0aa1-312e-422a-a521-a750541c3f10\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " Jul 08 03:32:30.450613 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.450575 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-istio-data\") pod \"e0bc0aa1-312e-422a-a521-a750541c3f10\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " Jul 08 03:32:30.450613 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.450600 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x82dd\" (UniqueName: \"kubernetes.io/projected/e0bc0aa1-312e-422a-a521-a750541c3f10-kube-api-access-x82dd\") pod \"e0bc0aa1-312e-422a-a521-a750541c3f10\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " Jul 08 03:32:30.450852 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.450629 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0bc0aa1-312e-422a-a521-a750541c3f10-istiod-ca-cert\") pod \"e0bc0aa1-312e-422a-a521-a750541c3f10\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " Jul 08 03:32:30.450852 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.450652 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-workload-socket\") pod \"e0bc0aa1-312e-422a-a521-a750541c3f10\" (UID: \"e0bc0aa1-312e-422a-a521-a750541c3f10\") " Jul 08 03:32:30.450852 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.450695 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e0bc0aa1-312e-422a-a521-a750541c3f10" (UID: "e0bc0aa1-312e-422a-a521-a750541c3f10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:32:30.450852 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.450838 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e0bc0aa1-312e-422a-a521-a750541c3f10" (UID: "e0bc0aa1-312e-422a-a521-a750541c3f10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:32:30.451131 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.451039 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-workload-certs\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:32:30.451131 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.451064 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-credential-socket\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:32:30.451131 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.451081 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e0bc0aa1-312e-422a-a521-a750541c3f10" (UID: "e0bc0aa1-312e-422a-a521-a750541c3f10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:32:30.451557 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.451535 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0bc0aa1-312e-422a-a521-a750541c3f10-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e0bc0aa1-312e-422a-a521-a750541c3f10" (UID: "e0bc0aa1-312e-422a-a521-a750541c3f10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:32:30.451966 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.451904 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-istio-data" (OuterVolumeSpecName: "istio-data") pod "e0bc0aa1-312e-422a-a521-a750541c3f10" (UID: "e0bc0aa1-312e-422a-a521-a750541c3f10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:32:30.452949 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.452907 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0bc0aa1-312e-422a-a521-a750541c3f10-istio-token" (OuterVolumeSpecName: "istio-token") pod "e0bc0aa1-312e-422a-a521-a750541c3f10" (UID: "e0bc0aa1-312e-422a-a521-a750541c3f10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:32:30.453084 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.453056 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e0bc0aa1-312e-422a-a521-a750541c3f10" (UID: "e0bc0aa1-312e-422a-a521-a750541c3f10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:32:30.453360 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.453331 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0bc0aa1-312e-422a-a521-a750541c3f10-kube-api-access-x82dd" (OuterVolumeSpecName: "kube-api-access-x82dd") pod "e0bc0aa1-312e-422a-a521-a750541c3f10" (UID: "e0bc0aa1-312e-422a-a521-a750541c3f10"). InnerVolumeSpecName "kube-api-access-x82dd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:32:30.453360 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.453334 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0bc0aa1-312e-422a-a521-a750541c3f10-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e0bc0aa1-312e-422a-a521-a750541c3f10" (UID: "e0bc0aa1-312e-422a-a521-a750541c3f10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 03:32:30.551897 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.551860 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0bc0aa1-312e-422a-a521-a750541c3f10-istio-podinfo\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:32:30.551897 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.551890 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-istio-envoy\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:32:30.551897 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.551899 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-istio-data\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:32:30.552172 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.551907 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x82dd\" (UniqueName: \"kubernetes.io/projected/e0bc0aa1-312e-422a-a521-a750541c3f10-kube-api-access-x82dd\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:32:30.552172 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.551953 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0bc0aa1-312e-422a-a521-a750541c3f10-istiod-ca-cert\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:32:30.552172 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.551963 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0bc0aa1-312e-422a-a521-a750541c3f10-workload-socket\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:32:30.552172 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:30.551970 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0bc0aa1-312e-422a-a521-a750541c3f10-istio-token\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:32:31.130276 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:31.130242 2564 generic.go:356] "Generic (PLEG): container finished" podID="e0bc0aa1-312e-422a-a521-a750541c3f10" containerID="5be5b2be41da608d0b39e2bfb6d2d86585de182370981b7f3f58ba241873cd7b" exitCode=0 Jul 08 03:32:31.130276 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:31.130280 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" event={"ID":"e0bc0aa1-312e-422a-a521-a750541c3f10","Type":"ContainerDied","Data":"5be5b2be41da608d0b39e2bfb6d2d86585de182370981b7f3f58ba241873cd7b"} Jul 08 03:32:31.130507 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:31.130293 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" Jul 08 03:32:31.130507 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:31.130318 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh" event={"ID":"e0bc0aa1-312e-422a-a521-a750541c3f10","Type":"ContainerDied","Data":"502393c562f79061974442d4b4f2180cb99dae06ca1fb0c2d01ab8e8c1489b6a"} Jul 08 03:32:31.130507 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:31.130339 2564 scope.go:117] "RemoveContainer" containerID="5be5b2be41da608d0b39e2bfb6d2d86585de182370981b7f3f58ba241873cd7b" Jul 08 03:32:31.142644 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:31.142617 2564 scope.go:117] "RemoveContainer" containerID="1175528ac563f3a68ea10b4ae39e048ea9ef1832de937302b2bcfcf69544d6d1" Jul 08 03:32:31.152022 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:31.151991 2564 scope.go:117] "RemoveContainer" containerID="5be5b2be41da608d0b39e2bfb6d2d86585de182370981b7f3f58ba241873cd7b" Jul 08 03:32:31.152370 ip-10-0-128-70 kubenswrapper[2564]: E0708 03:32:31.152348 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5be5b2be41da608d0b39e2bfb6d2d86585de182370981b7f3f58ba241873cd7b\": container with ID starting with 5be5b2be41da608d0b39e2bfb6d2d86585de182370981b7f3f58ba241873cd7b not found: ID does not exist" containerID="5be5b2be41da608d0b39e2bfb6d2d86585de182370981b7f3f58ba241873cd7b" Jul 08 03:32:31.152452 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:31.152379 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5be5b2be41da608d0b39e2bfb6d2d86585de182370981b7f3f58ba241873cd7b"} err="failed to get container status \"5be5b2be41da608d0b39e2bfb6d2d86585de182370981b7f3f58ba241873cd7b\": rpc error: code = NotFound desc = could not find container \"5be5b2be41da608d0b39e2bfb6d2d86585de182370981b7f3f58ba241873cd7b\": container with ID starting with 5be5b2be41da608d0b39e2bfb6d2d86585de182370981b7f3f58ba241873cd7b not found: ID does not exist" Jul 08 03:32:31.152452 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:31.152399 2564 scope.go:117] "RemoveContainer" containerID="1175528ac563f3a68ea10b4ae39e048ea9ef1832de937302b2bcfcf69544d6d1" Jul 08 03:32:31.152681 ip-10-0-128-70 kubenswrapper[2564]: E0708 03:32:31.152655 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1175528ac563f3a68ea10b4ae39e048ea9ef1832de937302b2bcfcf69544d6d1\": container with ID starting with 1175528ac563f3a68ea10b4ae39e048ea9ef1832de937302b2bcfcf69544d6d1 not found: ID does not exist" containerID="1175528ac563f3a68ea10b4ae39e048ea9ef1832de937302b2bcfcf69544d6d1" Jul 08 03:32:31.152727 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:31.152686 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1175528ac563f3a68ea10b4ae39e048ea9ef1832de937302b2bcfcf69544d6d1"} err="failed to get container status \"1175528ac563f3a68ea10b4ae39e048ea9ef1832de937302b2bcfcf69544d6d1\": rpc error: code = NotFound desc = could not find container \"1175528ac563f3a68ea10b4ae39e048ea9ef1832de937302b2bcfcf69544d6d1\": container with ID starting with 1175528ac563f3a68ea10b4ae39e048ea9ef1832de937302b2bcfcf69544d6d1 not found: ID does not exist" Jul 08 03:32:31.160091 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:31.160060 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh"] Jul 08 03:32:31.164223 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:31.164194 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1cf90851-openshift-default-775cb4d759-7lslh"] Jul 08 03:32:31.886577 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:31.886536 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0bc0aa1-312e-422a-a521-a750541c3f10" path="/var/lib/kubelet/pods/e0bc0aa1-312e-422a-a521-a750541c3f10/volumes" Jul 08 03:32:58.757558 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:32:58.757514 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:33:04.502873 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:33:04.502828 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:33:18.929338 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:33:18.929259 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:33:19.028243 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:33:19.028202 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:33:49.529121 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:33:49.529083 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:33:50.679528 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:33:50.679489 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:33:50.779552 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:33:50.779507 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:33:55.449589 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:33:55.449554 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:34:01.018164 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:01.018127 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:34:01.131108 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:01.131072 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:34:01.221757 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:01.221719 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:34:11.761931 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:11.761893 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:34:12.095451 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:12.095359 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:34:33.068067 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:33.068028 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:34:38.912416 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:38.912334 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:34:38.979176 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:38.979138 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:34:39.259451 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.259365 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:34:39.527099 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.527017 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8"] Jul 08 03:34:39.527428 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.527399 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43d77382-a8c8-47a9-a0cb-8248a6607338" containerName="main" Jul 08 03:34:39.527428 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.527422 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="43d77382-a8c8-47a9-a0cb-8248a6607338" containerName="main" Jul 08 03:34:39.527428 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.527445 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0bc0aa1-312e-422a-a521-a750541c3f10" containerName="istio-proxy" Jul 08 03:34:39.527428 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.527455 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0bc0aa1-312e-422a-a521-a750541c3f10" containerName="istio-proxy" Jul 08 03:34:39.527753 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.527542 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0bc0aa1-312e-422a-a521-a750541c3f10" containerName="istio-proxy" Jul 08 03:34:39.527753 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.527555 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0bc0aa1-312e-422a-a521-a750541c3f10" containerName="istio-proxy" Jul 08 03:34:39.527753 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.527566 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="43d77382-a8c8-47a9-a0cb-8248a6607338" containerName="main" Jul 08 03:34:39.531789 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.531763 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.535481 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.535458 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2515cc-openshift-default-dockercfg-lzjkt\"" Jul 08 03:34:39.555528 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.555497 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8"] Jul 08 03:34:39.557524 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.557497 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:34:39.616591 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.616547 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb7cx\" (UniqueName: \"kubernetes.io/projected/1979fbee-3299-4cee-a183-71d4a9cfec7d-kube-api-access-nb7cx\") pod \"e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.616794 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.616634 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1979fbee-3299-4cee-a183-71d4a9cfec7d-istio-token\") pod \"e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.616794 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.616660 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-workload-certs\") pod \"e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.616794 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.616682 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-istio-envoy\") pod \"e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.616794 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.616705 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1979fbee-3299-4cee-a183-71d4a9cfec7d-istiod-ca-cert\") pod \"e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.616794 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.616758 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1979fbee-3299-4cee-a183-71d4a9cfec7d-istio-podinfo\") pod \"e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.617077 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.616793 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-istio-data\") pod \"e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.617077 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.616841 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-workload-socket\") pod \"e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.617077 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.616894 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-credential-socket\") pod \"e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.654682 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.654644 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:34:39.718204 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.718163 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-workload-socket\") pod \"e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.718407 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.718225 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-credential-socket\") pod \"e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.718407 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.718265 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb7cx\" (UniqueName: \"kubernetes.io/projected/1979fbee-3299-4cee-a183-71d4a9cfec7d-kube-api-access-nb7cx\") pod \"e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.718407 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.718316 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1979fbee-3299-4cee-a183-71d4a9cfec7d-istio-token\") pod \"e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.718407 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.718341 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-workload-certs\") pod \"e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.718640 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.718439 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-istio-envoy\") pod \"e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.718640 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.718471 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1979fbee-3299-4cee-a183-71d4a9cfec7d-istiod-ca-cert\") pod \"e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.718640 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.718538 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1979fbee-3299-4cee-a183-71d4a9cfec7d-istio-podinfo\") pod \"e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.718640 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.718567 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-istio-data\") pod \"e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.718640 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.718624 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-workload-socket\") pod \"e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.718907 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.718669 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-credential-socket\") pod \"e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.718907 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.718775 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-workload-certs\") pod \"e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.719098 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.718977 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-istio-data\") pod \"e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.719227 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.719209 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1979fbee-3299-4cee-a183-71d4a9cfec7d-istiod-ca-cert\") pod \"e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.720867 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.720847 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-istio-envoy\") pod \"e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.721105 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.721087 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1979fbee-3299-4cee-a183-71d4a9cfec7d-istio-podinfo\") pod \"e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.727379 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.727355 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1979fbee-3299-4cee-a183-71d4a9cfec7d-istio-token\") pod \"e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.727955 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.727910 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb7cx\" (UniqueName: \"kubernetes.io/projected/1979fbee-3299-4cee-a183-71d4a9cfec7d-kube-api-access-nb7cx\") pod \"e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.844296 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.844188 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:39.984665 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.984636 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8"] Jul 08 03:34:39.986892 ip-10-0-128-70 kubenswrapper[2564]: W0708 03:34:39.986866 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1979fbee_3299_4cee_a183_71d4a9cfec7d.slice/crio-1941ead71b707e99041c0de8efcec6dfa27abbd935f8aaa6868c6d2933cbf15b WatchSource:0}: Error finding container 1941ead71b707e99041c0de8efcec6dfa27abbd935f8aaa6868c6d2933cbf15b: Status 404 returned error can't find the container with id 1941ead71b707e99041c0de8efcec6dfa27abbd935f8aaa6868c6d2933cbf15b Jul 08 03:34:39.989100 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.989068 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:34:39.989173 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.989142 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:34:39.989173 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:39.989169 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:34:40.737558 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:40.737522 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" event={"ID":"1979fbee-3299-4cee-a183-71d4a9cfec7d","Type":"ContainerStarted","Data":"43464bb3dbb7ba9ab18d60e55c3f5403a38bef0d7baa7928f58e1b8b3003c562"} Jul 08 03:34:40.737719 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:40.737571 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" event={"ID":"1979fbee-3299-4cee-a183-71d4a9cfec7d","Type":"ContainerStarted","Data":"1941ead71b707e99041c0de8efcec6dfa27abbd935f8aaa6868c6d2933cbf15b"} Jul 08 03:34:40.759198 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:40.759148 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" podStartSLOduration=1.7591298100000001 podStartE2EDuration="1.75912981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:34:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:34:40.757835942 +0000 UTC m=+2155.415211815" watchObservedRunningTime="2026-07-08 03:34:40.75912981 +0000 UTC m=+2155.416505682" Jul 08 03:34:40.845098 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:40.845046 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:41.849654 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:41.849627 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:42.671466 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:42.671430 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:34:42.747370 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:42.747334 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:42.748344 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:42.748325 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:42.799328 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:42.799291 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:34:42.935585 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:42.935486 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:34:43.372138 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:43.372049 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:34:43.470269 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:43.470233 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:34:47.172721 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:47.172683 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:34:47.484804 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:47.484721 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:34:47.796522 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:47.796428 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:34:48.219973 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:48.219911 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:34:54.368038 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:54.367954 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:34:54.443246 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:54.443201 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:34:54.523075 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:54.523039 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:34:54.623473 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:54.623382 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:34:54.942952 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:54.942907 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:34:55.044622 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:55.044592 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:34:58.535067 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:58.533798 2564 patch_prober.go:28] interesting pod/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": read tcp 10.134.0.2:50968->10.134.0.36:15021: read: connection reset by peer" start-of-body= Jul 08 03:34:58.535067 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:58.533871 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" podUID="1979fbee-3299-4cee-a183-71d4a9cfec7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": read tcp 10.134.0.2:50968->10.134.0.36:15021: read: connection reset by peer" Jul 08 03:34:58.832758 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:58.832663 2564 generic.go:356] "Generic (PLEG): container finished" podID="1979fbee-3299-4cee-a183-71d4a9cfec7d" containerID="43464bb3dbb7ba9ab18d60e55c3f5403a38bef0d7baa7928f58e1b8b3003c562" exitCode=0 Jul 08 03:34:58.832758 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:58.832737 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" event={"ID":"1979fbee-3299-4cee-a183-71d4a9cfec7d","Type":"ContainerDied","Data":"43464bb3dbb7ba9ab18d60e55c3f5403a38bef0d7baa7928f58e1b8b3003c562"} Jul 08 03:34:58.833187 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:58.833173 2564 scope.go:117] "RemoveContainer" containerID="43464bb3dbb7ba9ab18d60e55c3f5403a38bef0d7baa7928f58e1b8b3003c562" Jul 08 03:34:58.834271 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:58.834240 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:34:58.834337 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:58.834326 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:34:58.834378 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:58.834362 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:34:58.867287 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:58.867253 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:34:59.840423 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:59.840389 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" event={"ID":"1979fbee-3299-4cee-a183-71d4a9cfec7d","Type":"ContainerStarted","Data":"94a4f2614f556b78c513612907779610be36ca640d60cefb04c45a7d71c0acab"} Jul 08 03:34:59.844395 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:59.844375 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:34:59.844629 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:59.844609 2564 patch_prober.go:28] interesting pod/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 08 03:34:59.844677 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:34:59.844659 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" podUID="1979fbee-3299-4cee-a183-71d4a9cfec7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 08 03:35:00.845201 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:00.845170 2564 patch_prober.go:28] interesting pod/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 08 03:35:00.845606 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:00.845220 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" podUID="1979fbee-3299-4cee-a183-71d4a9cfec7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 08 03:35:01.845048 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:01.845006 2564 patch_prober.go:28] interesting pod/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 08 03:35:01.845260 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:01.845067 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" podUID="1979fbee-3299-4cee-a183-71d4a9cfec7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 08 03:35:02.845038 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:02.844998 2564 patch_prober.go:28] interesting pod/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 08 03:35:02.845239 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:02.845054 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" podUID="1979fbee-3299-4cee-a183-71d4a9cfec7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 08 03:35:03.848641 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:03.848608 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:35:03.849075 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:03.848994 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:35:03.849759 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:03.849739 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:35:04.105114 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:04.105035 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:35:04.205735 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:04.205698 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:35:11.318855 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.318816 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j"] Jul 08 03:35:11.319357 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.319334 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0bc0aa1-312e-422a-a521-a750541c3f10" containerName="istio-proxy" Jul 08 03:35:11.319436 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.319360 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0bc0aa1-312e-422a-a521-a750541c3f10" containerName="istio-proxy" Jul 08 03:35:11.322540 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.322522 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" Jul 08 03:35:11.325495 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.325464 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2515cc\"/\"test-model-2515cc-kserve-self-signed-certs\"" Jul 08 03:35:11.325495 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.325479 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2515cc\"/\"kube-root-ca.crt\"" Jul 08 03:35:11.325701 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.325499 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2515cc\"/\"default-dockercfg-tlprm\"" Jul 08 03:35:11.325701 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.325540 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2515cc\"/\"openshift-service-ca.crt\"" Jul 08 03:35:11.330726 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.330696 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j"] Jul 08 03:35:11.415826 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.415792 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:35:11.420003 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.419975 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b682100d-7267-4b2d-acef-bb78f37e1573-tls-certs\") pod \"test-model-2515cc-kserve-bb6797995-fc26j\" (UID: \"b682100d-7267-4b2d-acef-bb78f37e1573\") " pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" Jul 08 03:35:11.420141 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.420017 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b682100d-7267-4b2d-acef-bb78f37e1573-dshm\") pod \"test-model-2515cc-kserve-bb6797995-fc26j\" (UID: \"b682100d-7267-4b2d-acef-bb78f37e1573\") " pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" Jul 08 03:35:11.420141 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.420038 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b682100d-7267-4b2d-acef-bb78f37e1573-model-cache\") pod \"test-model-2515cc-kserve-bb6797995-fc26j\" (UID: \"b682100d-7267-4b2d-acef-bb78f37e1573\") " pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" Jul 08 03:35:11.420141 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.420072 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b682100d-7267-4b2d-acef-bb78f37e1573-home\") pod \"test-model-2515cc-kserve-bb6797995-fc26j\" (UID: \"b682100d-7267-4b2d-acef-bb78f37e1573\") " pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" Jul 08 03:35:11.420246 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.420179 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b682100d-7267-4b2d-acef-bb78f37e1573-tmp-dir\") pod \"test-model-2515cc-kserve-bb6797995-fc26j\" (UID: \"b682100d-7267-4b2d-acef-bb78f37e1573\") " pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" Jul 08 03:35:11.420246 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.420222 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-strgp\" (UniqueName: \"kubernetes.io/projected/b682100d-7267-4b2d-acef-bb78f37e1573-kube-api-access-strgp\") pod \"test-model-2515cc-kserve-bb6797995-fc26j\" (UID: \"b682100d-7267-4b2d-acef-bb78f37e1573\") " pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" Jul 08 03:35:11.521536 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.521494 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b682100d-7267-4b2d-acef-bb78f37e1573-tmp-dir\") pod \"test-model-2515cc-kserve-bb6797995-fc26j\" (UID: \"b682100d-7267-4b2d-acef-bb78f37e1573\") " pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" Jul 08 03:35:11.521792 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.521551 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-strgp\" (UniqueName: \"kubernetes.io/projected/b682100d-7267-4b2d-acef-bb78f37e1573-kube-api-access-strgp\") pod \"test-model-2515cc-kserve-bb6797995-fc26j\" (UID: \"b682100d-7267-4b2d-acef-bb78f37e1573\") " pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" Jul 08 03:35:11.521792 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.521582 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b682100d-7267-4b2d-acef-bb78f37e1573-tls-certs\") pod \"test-model-2515cc-kserve-bb6797995-fc26j\" (UID: \"b682100d-7267-4b2d-acef-bb78f37e1573\") " pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" Jul 08 03:35:11.521792 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.521610 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b682100d-7267-4b2d-acef-bb78f37e1573-dshm\") pod \"test-model-2515cc-kserve-bb6797995-fc26j\" (UID: \"b682100d-7267-4b2d-acef-bb78f37e1573\") " pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" Jul 08 03:35:11.521792 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.521629 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b682100d-7267-4b2d-acef-bb78f37e1573-model-cache\") pod \"test-model-2515cc-kserve-bb6797995-fc26j\" (UID: \"b682100d-7267-4b2d-acef-bb78f37e1573\") " pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" Jul 08 03:35:11.521792 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.521650 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b682100d-7267-4b2d-acef-bb78f37e1573-home\") pod \"test-model-2515cc-kserve-bb6797995-fc26j\" (UID: \"b682100d-7267-4b2d-acef-bb78f37e1573\") " pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" Jul 08 03:35:11.522103 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.522034 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b682100d-7267-4b2d-acef-bb78f37e1573-tmp-dir\") pod \"test-model-2515cc-kserve-bb6797995-fc26j\" (UID: \"b682100d-7267-4b2d-acef-bb78f37e1573\") " pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" Jul 08 03:35:11.522144 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.522108 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b682100d-7267-4b2d-acef-bb78f37e1573-home\") pod \"test-model-2515cc-kserve-bb6797995-fc26j\" (UID: \"b682100d-7267-4b2d-acef-bb78f37e1573\") " pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" Jul 08 03:35:11.522178 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.522158 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b682100d-7267-4b2d-acef-bb78f37e1573-model-cache\") pod \"test-model-2515cc-kserve-bb6797995-fc26j\" (UID: \"b682100d-7267-4b2d-acef-bb78f37e1573\") " pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" Jul 08 03:35:11.524218 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.524191 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b682100d-7267-4b2d-acef-bb78f37e1573-dshm\") pod \"test-model-2515cc-kserve-bb6797995-fc26j\" (UID: \"b682100d-7267-4b2d-acef-bb78f37e1573\") " pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" Jul 08 03:35:11.524339 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.524323 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b682100d-7267-4b2d-acef-bb78f37e1573-tls-certs\") pod \"test-model-2515cc-kserve-bb6797995-fc26j\" (UID: \"b682100d-7267-4b2d-acef-bb78f37e1573\") " pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" Jul 08 03:35:11.531435 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.531409 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-strgp\" (UniqueName: \"kubernetes.io/projected/b682100d-7267-4b2d-acef-bb78f37e1573-kube-api-access-strgp\") pod \"test-model-2515cc-kserve-bb6797995-fc26j\" (UID: \"b682100d-7267-4b2d-acef-bb78f37e1573\") " pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" Jul 08 03:35:11.632620 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.632585 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" Jul 08 03:35:11.772600 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.772524 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j"] Jul 08 03:35:11.775306 ip-10-0-128-70 kubenswrapper[2564]: W0708 03:35:11.775276 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb682100d_7267_4b2d_acef_bb78f37e1573.slice/crio-ae0505fa19cd3092300da78aeec2c029ed9238d54c57a1cbdc5080835f83c73a WatchSource:0}: Error finding container ae0505fa19cd3092300da78aeec2c029ed9238d54c57a1cbdc5080835f83c73a: Status 404 returned error can't find the container with id ae0505fa19cd3092300da78aeec2c029ed9238d54c57a1cbdc5080835f83c73a Jul 08 03:35:11.897528 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.897418 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" event={"ID":"b682100d-7267-4b2d-acef-bb78f37e1573","Type":"ContainerStarted","Data":"ed0d52806f69919fcc5e2d20b9e25a390d04f4245dbbe49d6f6cfe31f84cb0d9"} Jul 08 03:35:11.897528 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.897464 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" event={"ID":"b682100d-7267-4b2d-acef-bb78f37e1573","Type":"ContainerStarted","Data":"ae0505fa19cd3092300da78aeec2c029ed9238d54c57a1cbdc5080835f83c73a"} Jul 08 03:35:11.932508 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:11.932451 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" podStartSLOduration=0.932433357 podStartE2EDuration="932.433357ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:35:11.93036648 +0000 UTC m=+2186.587742346" watchObservedRunningTime="2026-07-08 03:35:11.932433357 +0000 UTC m=+2186.589809227" Jul 08 03:35:15.831566 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:15.831535 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:35:15.932254 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:15.932209 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:35:21.364686 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:21.364649 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:35:21.633405 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:21.633368 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" Jul 08 03:35:21.633592 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:21.633442 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" Jul 08 03:35:21.646088 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:21.646059 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" Jul 08 03:35:21.955903 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:21.955822 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" Jul 08 03:35:37.733957 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:37.733906 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv"] Jul 08 03:35:37.741059 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:37.738979 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" Jul 08 03:35:37.743236 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:37.743209 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7142f2\"/\"test-model-7142f2-kserve-self-signed-certs\"" Jul 08 03:35:37.743455 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:37.743210 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7142f2\"/\"default-dockercfg-vkpfp\"" Jul 08 03:35:37.743455 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:37.743293 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7142f2\"/\"openshift-service-ca.crt\"" Jul 08 03:35:37.743455 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:37.743320 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7142f2\"/\"kube-root-ca.crt\"" Jul 08 03:35:37.748077 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:37.748048 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv"] Jul 08 03:35:37.760753 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:37.760713 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:35:37.866491 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:37.866441 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0e2ec6b-d0b7-4366-811a-85e157284b45-dshm\") pod \"test-model-7142f2-kserve-68c8595cb5-6ndsv\" (UID: \"f0e2ec6b-d0b7-4366-811a-85e157284b45\") " pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" Jul 08 03:35:37.866682 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:37.866495 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0e2ec6b-d0b7-4366-811a-85e157284b45-tmp-dir\") pod \"test-model-7142f2-kserve-68c8595cb5-6ndsv\" (UID: \"f0e2ec6b-d0b7-4366-811a-85e157284b45\") " pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" Jul 08 03:35:37.866682 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:37.866552 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0e2ec6b-d0b7-4366-811a-85e157284b45-tls-certs\") pod \"test-model-7142f2-kserve-68c8595cb5-6ndsv\" (UID: \"f0e2ec6b-d0b7-4366-811a-85e157284b45\") " pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" Jul 08 03:35:37.866682 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:37.866645 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0e2ec6b-d0b7-4366-811a-85e157284b45-model-cache\") pod \"test-model-7142f2-kserve-68c8595cb5-6ndsv\" (UID: \"f0e2ec6b-d0b7-4366-811a-85e157284b45\") " pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" Jul 08 03:35:37.866682 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:37.866674 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0e2ec6b-d0b7-4366-811a-85e157284b45-home\") pod \"test-model-7142f2-kserve-68c8595cb5-6ndsv\" (UID: \"f0e2ec6b-d0b7-4366-811a-85e157284b45\") " pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" Jul 08 03:35:37.866855 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:37.866702 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gpw5\" (UniqueName: \"kubernetes.io/projected/f0e2ec6b-d0b7-4366-811a-85e157284b45-kube-api-access-9gpw5\") pod \"test-model-7142f2-kserve-68c8595cb5-6ndsv\" (UID: \"f0e2ec6b-d0b7-4366-811a-85e157284b45\") " pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" Jul 08 03:35:37.967395 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:37.967356 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0e2ec6b-d0b7-4366-811a-85e157284b45-dshm\") pod \"test-model-7142f2-kserve-68c8595cb5-6ndsv\" (UID: \"f0e2ec6b-d0b7-4366-811a-85e157284b45\") " pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" Jul 08 03:35:37.967395 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:37.967400 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0e2ec6b-d0b7-4366-811a-85e157284b45-tmp-dir\") pod \"test-model-7142f2-kserve-68c8595cb5-6ndsv\" (UID: \"f0e2ec6b-d0b7-4366-811a-85e157284b45\") " pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" Jul 08 03:35:37.967648 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:37.967432 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0e2ec6b-d0b7-4366-811a-85e157284b45-tls-certs\") pod \"test-model-7142f2-kserve-68c8595cb5-6ndsv\" (UID: \"f0e2ec6b-d0b7-4366-811a-85e157284b45\") " pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" Jul 08 03:35:37.967648 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:37.967480 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0e2ec6b-d0b7-4366-811a-85e157284b45-model-cache\") pod \"test-model-7142f2-kserve-68c8595cb5-6ndsv\" (UID: \"f0e2ec6b-d0b7-4366-811a-85e157284b45\") " pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" Jul 08 03:35:37.967648 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:37.967501 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0e2ec6b-d0b7-4366-811a-85e157284b45-home\") pod \"test-model-7142f2-kserve-68c8595cb5-6ndsv\" (UID: \"f0e2ec6b-d0b7-4366-811a-85e157284b45\") " pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" Jul 08 03:35:37.967648 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:37.967624 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gpw5\" (UniqueName: \"kubernetes.io/projected/f0e2ec6b-d0b7-4366-811a-85e157284b45-kube-api-access-9gpw5\") pod \"test-model-7142f2-kserve-68c8595cb5-6ndsv\" (UID: \"f0e2ec6b-d0b7-4366-811a-85e157284b45\") " pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" Jul 08 03:35:37.967987 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:37.967963 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0e2ec6b-d0b7-4366-811a-85e157284b45-model-cache\") pod \"test-model-7142f2-kserve-68c8595cb5-6ndsv\" (UID: \"f0e2ec6b-d0b7-4366-811a-85e157284b45\") " pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" Jul 08 03:35:37.968071 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:37.968018 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0e2ec6b-d0b7-4366-811a-85e157284b45-home\") pod \"test-model-7142f2-kserve-68c8595cb5-6ndsv\" (UID: \"f0e2ec6b-d0b7-4366-811a-85e157284b45\") " pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" Jul 08 03:35:37.968071 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:37.968036 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0e2ec6b-d0b7-4366-811a-85e157284b45-tmp-dir\") pod \"test-model-7142f2-kserve-68c8595cb5-6ndsv\" (UID: \"f0e2ec6b-d0b7-4366-811a-85e157284b45\") " pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" Jul 08 03:35:37.969790 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:37.969769 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0e2ec6b-d0b7-4366-811a-85e157284b45-dshm\") pod \"test-model-7142f2-kserve-68c8595cb5-6ndsv\" (UID: \"f0e2ec6b-d0b7-4366-811a-85e157284b45\") " pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" Jul 08 03:35:37.970138 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:37.970119 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0e2ec6b-d0b7-4366-811a-85e157284b45-tls-certs\") pod \"test-model-7142f2-kserve-68c8595cb5-6ndsv\" (UID: \"f0e2ec6b-d0b7-4366-811a-85e157284b45\") " pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" Jul 08 03:35:37.975894 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:37.975865 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gpw5\" (UniqueName: \"kubernetes.io/projected/f0e2ec6b-d0b7-4366-811a-85e157284b45-kube-api-access-9gpw5\") pod \"test-model-7142f2-kserve-68c8595cb5-6ndsv\" (UID: \"f0e2ec6b-d0b7-4366-811a-85e157284b45\") " pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" Jul 08 03:35:38.052829 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:38.052731 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" Jul 08 03:35:38.193934 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:38.193882 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv"] Jul 08 03:35:38.197615 ip-10-0-128-70 kubenswrapper[2564]: W0708 03:35:38.197584 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0e2ec6b_d0b7_4366_811a_85e157284b45.slice/crio-fc02e3b75558fd2925176cae1bf1509f307531af18ab1dd9de3d1270ae39bad0 WatchSource:0}: Error finding container fc02e3b75558fd2925176cae1bf1509f307531af18ab1dd9de3d1270ae39bad0: Status 404 returned error can't find the container with id fc02e3b75558fd2925176cae1bf1509f307531af18ab1dd9de3d1270ae39bad0 Jul 08 03:35:39.028390 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:39.028351 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" event={"ID":"f0e2ec6b-d0b7-4366-811a-85e157284b45","Type":"ContainerStarted","Data":"dee1350683cbfbcb729fdddf89d6114eaf72ba48f5a17598c3f74a744388d58e"} Jul 08 03:35:39.028804 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:39.028400 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" event={"ID":"f0e2ec6b-d0b7-4366-811a-85e157284b45","Type":"ContainerStarted","Data":"fc02e3b75558fd2925176cae1bf1509f307531af18ab1dd9de3d1270ae39bad0"} Jul 08 03:35:39.045577 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:39.045522 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" podStartSLOduration=2.04550485 podStartE2EDuration="2.04550485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:35:39.04477193 +0000 UTC m=+2213.702147802" watchObservedRunningTime="2026-07-08 03:35:39.04550485 +0000 UTC m=+2213.702880721" Jul 08 03:35:39.119646 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:39.119598 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:35:39.220852 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:39.220813 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:35:48.053407 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:48.053350 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" Jul 08 03:35:48.053911 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:48.053429 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" Jul 08 03:35:48.066253 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:48.066221 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" Jul 08 03:35:48.099379 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:48.099345 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" Jul 08 03:35:56.807561 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:56.807524 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:35:57.108606 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:57.108512 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:35:57.495899 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:57.495864 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:35:59.737320 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:59.737287 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:35:59.830046 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:59.830007 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:35:59.998385 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:35:59.998304 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:36:00.153360 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:36:00.153324 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:36:00.658667 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:36:00.658621 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:36:00.755236 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:36:00.755178 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:36:04.225307 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:36:04.225268 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:36:29.767666 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:36:29.767577 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:36:29.838988 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:36:29.838952 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:36:33.726694 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:36:33.726653 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:36:46.497969 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:36:46.497906 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:36:46.870757 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:36:46.870653 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:36:46.927410 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:36:46.927366 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:36:51.944663 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:36:51.944626 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:36:52.466494 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:36:52.466450 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:37:29.071975 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:29.071937 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:37:30.124078 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:30.124041 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:37:35.008032 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:35.007987 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:37:35.078260 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:35.078219 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:37:35.300319 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:35.300231 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:37:36.548293 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:36.548206 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j"] Jul 08 03:37:36.568642 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:36.568602 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:37:37.483267 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:37.483227 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv"] Jul 08 03:37:37.534955 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:37.534905 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:37:37.726746 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:37.726709 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:37:38.063452 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:38.063419 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:37:38.193964 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:38.193912 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:37:38.294579 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:38.294541 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:37:38.472344 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:38.472314 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:37:43.141677 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:43.141639 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:37:51.569227 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:51.569171 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" podUID="b682100d-7267-4b2d-acef-bb78f37e1573" containerName="main" containerID="cri-o://ed0d52806f69919fcc5e2d20b9e25a390d04f4245dbbe49d6f6cfe31f84cb0d9" gracePeriod=45 Jul 08 03:37:51.829659 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:51.829590 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" Jul 08 03:37:51.889541 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:51.889516 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-strgp\" (UniqueName: \"kubernetes.io/projected/b682100d-7267-4b2d-acef-bb78f37e1573-kube-api-access-strgp\") pod \"b682100d-7267-4b2d-acef-bb78f37e1573\" (UID: \"b682100d-7267-4b2d-acef-bb78f37e1573\") " Jul 08 03:37:51.889718 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:51.889549 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b682100d-7267-4b2d-acef-bb78f37e1573-tls-certs\") pod \"b682100d-7267-4b2d-acef-bb78f37e1573\" (UID: \"b682100d-7267-4b2d-acef-bb78f37e1573\") " Jul 08 03:37:51.889718 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:51.889580 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b682100d-7267-4b2d-acef-bb78f37e1573-home\") pod \"b682100d-7267-4b2d-acef-bb78f37e1573\" (UID: \"b682100d-7267-4b2d-acef-bb78f37e1573\") " Jul 08 03:37:51.889718 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:51.889659 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b682100d-7267-4b2d-acef-bb78f37e1573-dshm\") pod \"b682100d-7267-4b2d-acef-bb78f37e1573\" (UID: \"b682100d-7267-4b2d-acef-bb78f37e1573\") " Jul 08 03:37:51.889718 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:51.889685 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b682100d-7267-4b2d-acef-bb78f37e1573-model-cache\") pod \"b682100d-7267-4b2d-acef-bb78f37e1573\" (UID: \"b682100d-7267-4b2d-acef-bb78f37e1573\") " Jul 08 03:37:51.889718 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:51.889699 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b682100d-7267-4b2d-acef-bb78f37e1573-tmp-dir\") pod \"b682100d-7267-4b2d-acef-bb78f37e1573\" (UID: \"b682100d-7267-4b2d-acef-bb78f37e1573\") " Jul 08 03:37:51.890038 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:51.889980 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b682100d-7267-4b2d-acef-bb78f37e1573-model-cache" (OuterVolumeSpecName: "model-cache") pod "b682100d-7267-4b2d-acef-bb78f37e1573" (UID: "b682100d-7267-4b2d-acef-bb78f37e1573"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:37:51.890038 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:51.889997 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b682100d-7267-4b2d-acef-bb78f37e1573-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b682100d-7267-4b2d-acef-bb78f37e1573" (UID: "b682100d-7267-4b2d-acef-bb78f37e1573"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:37:51.890038 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:51.890008 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b682100d-7267-4b2d-acef-bb78f37e1573-home" (OuterVolumeSpecName: "home") pod "b682100d-7267-4b2d-acef-bb78f37e1573" (UID: "b682100d-7267-4b2d-acef-bb78f37e1573"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:37:51.891996 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:51.891962 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b682100d-7267-4b2d-acef-bb78f37e1573-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b682100d-7267-4b2d-acef-bb78f37e1573" (UID: "b682100d-7267-4b2d-acef-bb78f37e1573"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:37:51.891996 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:51.891970 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b682100d-7267-4b2d-acef-bb78f37e1573-dshm" (OuterVolumeSpecName: "dshm") pod "b682100d-7267-4b2d-acef-bb78f37e1573" (UID: "b682100d-7267-4b2d-acef-bb78f37e1573"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:37:51.892124 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:51.892011 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b682100d-7267-4b2d-acef-bb78f37e1573-kube-api-access-strgp" (OuterVolumeSpecName: "kube-api-access-strgp") pod "b682100d-7267-4b2d-acef-bb78f37e1573" (UID: "b682100d-7267-4b2d-acef-bb78f37e1573"). InnerVolumeSpecName "kube-api-access-strgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:37:51.991374 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:51.991335 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b682100d-7267-4b2d-acef-bb78f37e1573-dshm\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:37:51.991374 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:51.991367 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b682100d-7267-4b2d-acef-bb78f37e1573-model-cache\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:37:51.991575 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:51.991395 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b682100d-7267-4b2d-acef-bb78f37e1573-tmp-dir\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:37:51.991575 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:51.991404 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-strgp\" (UniqueName: \"kubernetes.io/projected/b682100d-7267-4b2d-acef-bb78f37e1573-kube-api-access-strgp\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:37:51.991575 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:51.991413 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b682100d-7267-4b2d-acef-bb78f37e1573-tls-certs\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:37:51.991575 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:51.991421 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b682100d-7267-4b2d-acef-bb78f37e1573-home\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:37:52.503806 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.503735 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" podUID="f0e2ec6b-d0b7-4366-811a-85e157284b45" containerName="main" containerID="cri-o://dee1350683cbfbcb729fdddf89d6114eaf72ba48f5a17598c3f74a744388d58e" gracePeriod=45 Jul 08 03:37:52.687884 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.687852 2564 generic.go:356] "Generic (PLEG): container finished" podID="b682100d-7267-4b2d-acef-bb78f37e1573" containerID="ed0d52806f69919fcc5e2d20b9e25a390d04f4245dbbe49d6f6cfe31f84cb0d9" exitCode=0 Jul 08 03:37:52.688336 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.687943 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" Jul 08 03:37:52.688336 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.687951 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" event={"ID":"b682100d-7267-4b2d-acef-bb78f37e1573","Type":"ContainerDied","Data":"ed0d52806f69919fcc5e2d20b9e25a390d04f4245dbbe49d6f6cfe31f84cb0d9"} Jul 08 03:37:52.688336 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.688008 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j" event={"ID":"b682100d-7267-4b2d-acef-bb78f37e1573","Type":"ContainerDied","Data":"ae0505fa19cd3092300da78aeec2c029ed9238d54c57a1cbdc5080835f83c73a"} Jul 08 03:37:52.688336 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.688034 2564 scope.go:117] "RemoveContainer" containerID="ed0d52806f69919fcc5e2d20b9e25a390d04f4245dbbe49d6f6cfe31f84cb0d9" Jul 08 03:37:52.689952 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.689905 2564 generic.go:356] "Generic (PLEG): container finished" podID="f0e2ec6b-d0b7-4366-811a-85e157284b45" containerID="dee1350683cbfbcb729fdddf89d6114eaf72ba48f5a17598c3f74a744388d58e" exitCode=0 Jul 08 03:37:52.690061 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.689953 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" event={"ID":"f0e2ec6b-d0b7-4366-811a-85e157284b45","Type":"ContainerDied","Data":"dee1350683cbfbcb729fdddf89d6114eaf72ba48f5a17598c3f74a744388d58e"} Jul 08 03:37:52.698951 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.698931 2564 scope.go:117] "RemoveContainer" containerID="ed0d52806f69919fcc5e2d20b9e25a390d04f4245dbbe49d6f6cfe31f84cb0d9" Jul 08 03:37:52.699303 ip-10-0-128-70 kubenswrapper[2564]: E0708 03:37:52.699271 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed0d52806f69919fcc5e2d20b9e25a390d04f4245dbbe49d6f6cfe31f84cb0d9\": container with ID starting with ed0d52806f69919fcc5e2d20b9e25a390d04f4245dbbe49d6f6cfe31f84cb0d9 not found: ID does not exist" containerID="ed0d52806f69919fcc5e2d20b9e25a390d04f4245dbbe49d6f6cfe31f84cb0d9" Jul 08 03:37:52.699418 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.699315 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed0d52806f69919fcc5e2d20b9e25a390d04f4245dbbe49d6f6cfe31f84cb0d9"} err="failed to get container status \"ed0d52806f69919fcc5e2d20b9e25a390d04f4245dbbe49d6f6cfe31f84cb0d9\": rpc error: code = NotFound desc = could not find container \"ed0d52806f69919fcc5e2d20b9e25a390d04f4245dbbe49d6f6cfe31f84cb0d9\": container with ID starting with ed0d52806f69919fcc5e2d20b9e25a390d04f4245dbbe49d6f6cfe31f84cb0d9 not found: ID does not exist" Jul 08 03:37:52.713354 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.713324 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j"] Jul 08 03:37:52.721775 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.721745 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2515cc/test-model-2515cc-kserve-bb6797995-fc26j"] Jul 08 03:37:52.761275 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.761209 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" Jul 08 03:37:52.798579 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.798548 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0e2ec6b-d0b7-4366-811a-85e157284b45-tmp-dir\") pod \"f0e2ec6b-d0b7-4366-811a-85e157284b45\" (UID: \"f0e2ec6b-d0b7-4366-811a-85e157284b45\") " Jul 08 03:37:52.798731 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.798586 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0e2ec6b-d0b7-4366-811a-85e157284b45-tls-certs\") pod \"f0e2ec6b-d0b7-4366-811a-85e157284b45\" (UID: \"f0e2ec6b-d0b7-4366-811a-85e157284b45\") " Jul 08 03:37:52.798731 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.798609 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0e2ec6b-d0b7-4366-811a-85e157284b45-home\") pod \"f0e2ec6b-d0b7-4366-811a-85e157284b45\" (UID: \"f0e2ec6b-d0b7-4366-811a-85e157284b45\") " Jul 08 03:37:52.798731 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.798644 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0e2ec6b-d0b7-4366-811a-85e157284b45-dshm\") pod \"f0e2ec6b-d0b7-4366-811a-85e157284b45\" (UID: \"f0e2ec6b-d0b7-4366-811a-85e157284b45\") " Jul 08 03:37:52.798731 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.798687 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gpw5\" (UniqueName: \"kubernetes.io/projected/f0e2ec6b-d0b7-4366-811a-85e157284b45-kube-api-access-9gpw5\") pod \"f0e2ec6b-d0b7-4366-811a-85e157284b45\" (UID: \"f0e2ec6b-d0b7-4366-811a-85e157284b45\") " Jul 08 03:37:52.799007 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.798731 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0e2ec6b-d0b7-4366-811a-85e157284b45-model-cache\") pod \"f0e2ec6b-d0b7-4366-811a-85e157284b45\" (UID: \"f0e2ec6b-d0b7-4366-811a-85e157284b45\") " Jul 08 03:37:52.799007 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.798864 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e2ec6b-d0b7-4366-811a-85e157284b45-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f0e2ec6b-d0b7-4366-811a-85e157284b45" (UID: "f0e2ec6b-d0b7-4366-811a-85e157284b45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:37:52.799007 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.798878 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e2ec6b-d0b7-4366-811a-85e157284b45-home" (OuterVolumeSpecName: "home") pod "f0e2ec6b-d0b7-4366-811a-85e157284b45" (UID: "f0e2ec6b-d0b7-4366-811a-85e157284b45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:37:52.799179 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.799071 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e2ec6b-d0b7-4366-811a-85e157284b45-model-cache" (OuterVolumeSpecName: "model-cache") pod "f0e2ec6b-d0b7-4366-811a-85e157284b45" (UID: "f0e2ec6b-d0b7-4366-811a-85e157284b45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:37:52.799179 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.799110 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0e2ec6b-d0b7-4366-811a-85e157284b45-tmp-dir\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:37:52.799179 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.799123 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0e2ec6b-d0b7-4366-811a-85e157284b45-home\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:37:52.801647 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.801621 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0e2ec6b-d0b7-4366-811a-85e157284b45-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f0e2ec6b-d0b7-4366-811a-85e157284b45" (UID: "f0e2ec6b-d0b7-4366-811a-85e157284b45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 03:37:52.801647 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.801627 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e2ec6b-d0b7-4366-811a-85e157284b45-kube-api-access-9gpw5" (OuterVolumeSpecName: "kube-api-access-9gpw5") pod "f0e2ec6b-d0b7-4366-811a-85e157284b45" (UID: "f0e2ec6b-d0b7-4366-811a-85e157284b45"). InnerVolumeSpecName "kube-api-access-9gpw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:37:52.801782 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.801627 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e2ec6b-d0b7-4366-811a-85e157284b45-dshm" (OuterVolumeSpecName: "dshm") pod "f0e2ec6b-d0b7-4366-811a-85e157284b45" (UID: "f0e2ec6b-d0b7-4366-811a-85e157284b45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:37:52.899845 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.899802 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gpw5\" (UniqueName: \"kubernetes.io/projected/f0e2ec6b-d0b7-4366-811a-85e157284b45-kube-api-access-9gpw5\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:37:52.899845 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.899839 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0e2ec6b-d0b7-4366-811a-85e157284b45-model-cache\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:37:52.899845 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.899848 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0e2ec6b-d0b7-4366-811a-85e157284b45-tls-certs\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:37:52.899845 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:52.899855 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0e2ec6b-d0b7-4366-811a-85e157284b45-dshm\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:37:53.697038 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:53.697007 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" Jul 08 03:37:53.697038 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:53.697021 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv" event={"ID":"f0e2ec6b-d0b7-4366-811a-85e157284b45","Type":"ContainerDied","Data":"fc02e3b75558fd2925176cae1bf1509f307531af18ab1dd9de3d1270ae39bad0"} Jul 08 03:37:53.697574 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:53.697066 2564 scope.go:117] "RemoveContainer" containerID="dee1350683cbfbcb729fdddf89d6114eaf72ba48f5a17598c3f74a744388d58e" Jul 08 03:37:53.720276 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:53.720251 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv"] Jul 08 03:37:53.727247 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:53.727220 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-7142f2/test-model-7142f2-kserve-68c8595cb5-6ndsv"] Jul 08 03:37:53.886194 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:53.886165 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b682100d-7267-4b2d-acef-bb78f37e1573" path="/var/lib/kubelet/pods/b682100d-7267-4b2d-acef-bb78f37e1573/volumes" Jul 08 03:37:53.886554 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:37:53.886541 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0e2ec6b-d0b7-4366-811a-85e157284b45" path="/var/lib/kubelet/pods/f0e2ec6b-d0b7-4366-811a-85e157284b45/volumes" Jul 08 03:38:19.978873 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:19.978823 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:20.227699 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:20.227665 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8"] Jul 08 03:38:20.227915 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:20.227892 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" podUID="1979fbee-3299-4cee-a183-71d4a9cfec7d" containerName="istio-proxy" containerID="cri-o://94a4f2614f556b78c513612907779610be36ca640d60cefb04c45a7d71c0acab" gracePeriod=30 Jul 08 03:38:20.244592 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:20.244463 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:20.645211 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:20.645175 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:21.055224 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:21.055121 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:21.156804 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:21.156771 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:21.256762 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:21.256715 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:21.431653 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:21.431618 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:25.482679 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.482654 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:38:25.515091 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.515003 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1979fbee-3299-4cee-a183-71d4a9cfec7d-istio-token\") pod \"1979fbee-3299-4cee-a183-71d4a9cfec7d\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " Jul 08 03:38:25.515091 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.515049 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-istio-envoy\") pod \"1979fbee-3299-4cee-a183-71d4a9cfec7d\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " Jul 08 03:38:25.515091 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.515067 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1979fbee-3299-4cee-a183-71d4a9cfec7d-istiod-ca-cert\") pod \"1979fbee-3299-4cee-a183-71d4a9cfec7d\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " Jul 08 03:38:25.515091 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.515091 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-workload-certs\") pod \"1979fbee-3299-4cee-a183-71d4a9cfec7d\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " Jul 08 03:38:25.515423 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.515129 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-credential-socket\") pod \"1979fbee-3299-4cee-a183-71d4a9cfec7d\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " Jul 08 03:38:25.515423 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.515155 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1979fbee-3299-4cee-a183-71d4a9cfec7d-istio-podinfo\") pod \"1979fbee-3299-4cee-a183-71d4a9cfec7d\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " Jul 08 03:38:25.515423 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.515187 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-workload-socket\") pod \"1979fbee-3299-4cee-a183-71d4a9cfec7d\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " Jul 08 03:38:25.515423 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.515212 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-istio-data\") pod \"1979fbee-3299-4cee-a183-71d4a9cfec7d\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " Jul 08 03:38:25.515423 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.515269 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nb7cx\" (UniqueName: \"kubernetes.io/projected/1979fbee-3299-4cee-a183-71d4a9cfec7d-kube-api-access-nb7cx\") pod \"1979fbee-3299-4cee-a183-71d4a9cfec7d\" (UID: \"1979fbee-3299-4cee-a183-71d4a9cfec7d\") " Jul 08 03:38:25.515673 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.515449 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1979fbee-3299-4cee-a183-71d4a9cfec7d" (UID: "1979fbee-3299-4cee-a183-71d4a9cfec7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:38:25.515673 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.515591 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1979fbee-3299-4cee-a183-71d4a9cfec7d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1979fbee-3299-4cee-a183-71d4a9cfec7d" (UID: "1979fbee-3299-4cee-a183-71d4a9cfec7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:38:25.515970 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.515673 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1979fbee-3299-4cee-a183-71d4a9cfec7d-istiod-ca-cert\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:38:25.515970 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.515690 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-workload-certs\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:38:25.516087 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.515987 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1979fbee-3299-4cee-a183-71d4a9cfec7d" (UID: "1979fbee-3299-4cee-a183-71d4a9cfec7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:38:25.516087 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.516023 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1979fbee-3299-4cee-a183-71d4a9cfec7d" (UID: "1979fbee-3299-4cee-a183-71d4a9cfec7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:38:25.516541 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.516512 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-istio-data" (OuterVolumeSpecName: "istio-data") pod "1979fbee-3299-4cee-a183-71d4a9cfec7d" (UID: "1979fbee-3299-4cee-a183-71d4a9cfec7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:38:25.517576 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.517548 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1979fbee-3299-4cee-a183-71d4a9cfec7d" (UID: "1979fbee-3299-4cee-a183-71d4a9cfec7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:38:25.517714 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.517637 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1979fbee-3299-4cee-a183-71d4a9cfec7d-kube-api-access-nb7cx" (OuterVolumeSpecName: "kube-api-access-nb7cx") pod "1979fbee-3299-4cee-a183-71d4a9cfec7d" (UID: "1979fbee-3299-4cee-a183-71d4a9cfec7d"). InnerVolumeSpecName "kube-api-access-nb7cx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:38:25.518002 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.517978 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1979fbee-3299-4cee-a183-71d4a9cfec7d-istio-token" (OuterVolumeSpecName: "istio-token") pod "1979fbee-3299-4cee-a183-71d4a9cfec7d" (UID: "1979fbee-3299-4cee-a183-71d4a9cfec7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:38:25.518091 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.518050 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1979fbee-3299-4cee-a183-71d4a9cfec7d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1979fbee-3299-4cee-a183-71d4a9cfec7d" (UID: "1979fbee-3299-4cee-a183-71d4a9cfec7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 03:38:25.616752 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.616696 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1979fbee-3299-4cee-a183-71d4a9cfec7d-istio-token\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:38:25.616752 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.616742 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-istio-envoy\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:38:25.616752 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.616752 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-credential-socket\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:38:25.616752 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.616762 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1979fbee-3299-4cee-a183-71d4a9cfec7d-istio-podinfo\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:38:25.616752 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.616771 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-workload-socket\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:38:25.617102 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.616779 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1979fbee-3299-4cee-a183-71d4a9cfec7d-istio-data\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:38:25.617102 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.616787 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb7cx\" (UniqueName: \"kubernetes.io/projected/1979fbee-3299-4cee-a183-71d4a9cfec7d-kube-api-access-nb7cx\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:38:25.856707 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.856616 2564 generic.go:356] "Generic (PLEG): container finished" podID="1979fbee-3299-4cee-a183-71d4a9cfec7d" containerID="94a4f2614f556b78c513612907779610be36ca640d60cefb04c45a7d71c0acab" exitCode=0 Jul 08 03:38:25.856707 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.856661 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" Jul 08 03:38:25.856707 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.856693 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" event={"ID":"1979fbee-3299-4cee-a183-71d4a9cfec7d","Type":"ContainerDied","Data":"94a4f2614f556b78c513612907779610be36ca640d60cefb04c45a7d71c0acab"} Jul 08 03:38:25.856967 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.856739 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8" event={"ID":"1979fbee-3299-4cee-a183-71d4a9cfec7d","Type":"ContainerDied","Data":"1941ead71b707e99041c0de8efcec6dfa27abbd935f8aaa6868c6d2933cbf15b"} Jul 08 03:38:25.856967 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.856763 2564 scope.go:117] "RemoveContainer" containerID="94a4f2614f556b78c513612907779610be36ca640d60cefb04c45a7d71c0acab" Jul 08 03:38:25.866481 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.866460 2564 scope.go:117] "RemoveContainer" containerID="43464bb3dbb7ba9ab18d60e55c3f5403a38bef0d7baa7928f58e1b8b3003c562" Jul 08 03:38:25.874596 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.874577 2564 scope.go:117] "RemoveContainer" containerID="94a4f2614f556b78c513612907779610be36ca640d60cefb04c45a7d71c0acab" Jul 08 03:38:25.874840 ip-10-0-128-70 kubenswrapper[2564]: E0708 03:38:25.874822 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94a4f2614f556b78c513612907779610be36ca640d60cefb04c45a7d71c0acab\": container with ID starting with 94a4f2614f556b78c513612907779610be36ca640d60cefb04c45a7d71c0acab not found: ID does not exist" containerID="94a4f2614f556b78c513612907779610be36ca640d60cefb04c45a7d71c0acab" Jul 08 03:38:25.874894 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.874855 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a4f2614f556b78c513612907779610be36ca640d60cefb04c45a7d71c0acab"} err="failed to get container status \"94a4f2614f556b78c513612907779610be36ca640d60cefb04c45a7d71c0acab\": rpc error: code = NotFound desc = could not find container \"94a4f2614f556b78c513612907779610be36ca640d60cefb04c45a7d71c0acab\": container with ID starting with 94a4f2614f556b78c513612907779610be36ca640d60cefb04c45a7d71c0acab not found: ID does not exist" Jul 08 03:38:25.874894 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.874873 2564 scope.go:117] "RemoveContainer" containerID="43464bb3dbb7ba9ab18d60e55c3f5403a38bef0d7baa7928f58e1b8b3003c562" Jul 08 03:38:25.875180 ip-10-0-128-70 kubenswrapper[2564]: E0708 03:38:25.875156 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43464bb3dbb7ba9ab18d60e55c3f5403a38bef0d7baa7928f58e1b8b3003c562\": container with ID starting with 43464bb3dbb7ba9ab18d60e55c3f5403a38bef0d7baa7928f58e1b8b3003c562 not found: ID does not exist" containerID="43464bb3dbb7ba9ab18d60e55c3f5403a38bef0d7baa7928f58e1b8b3003c562" Jul 08 03:38:25.875238 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.875190 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43464bb3dbb7ba9ab18d60e55c3f5403a38bef0d7baa7928f58e1b8b3003c562"} err="failed to get container status \"43464bb3dbb7ba9ab18d60e55c3f5403a38bef0d7baa7928f58e1b8b3003c562\": rpc error: code = NotFound desc = could not find container \"43464bb3dbb7ba9ab18d60e55c3f5403a38bef0d7baa7928f58e1b8b3003c562\": container with ID starting with 43464bb3dbb7ba9ab18d60e55c3f5403a38bef0d7baa7928f58e1b8b3003c562 not found: ID does not exist" Jul 08 03:38:25.893981 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.893949 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8"] Jul 08 03:38:25.899199 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:25.899172 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2515cc-openshift-default-6f959bd7ff-v2rj8"] Jul 08 03:38:27.162667 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:27.162633 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:27.261856 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:27.261815 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:27.438769 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:27.438666 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:27.886155 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:27.886124 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1979fbee-3299-4cee-a183-71d4a9cfec7d" path="/var/lib/kubelet/pods/1979fbee-3299-4cee-a183-71d4a9cfec7d/volumes" Jul 08 03:38:27.963794 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:27.963756 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:28.292136 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:28.292052 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:28.686973 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:28.686909 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:31.551956 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:31.551888 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:31.643445 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:31.643412 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:31.714234 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:31.714199 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:31.816772 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:31.816677 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:32.129265 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:32.129232 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:32.244631 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:32.244597 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:36.082146 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:36.082105 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:36.491235 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:36.491197 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:36.833579 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:36.833489 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:36.968677 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:36.968640 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:39.646774 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:39.646742 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:39.757175 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:39.757144 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:39.859051 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:39.859018 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:39.990416 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:39.990334 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:40.414976 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:40.414947 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:40.517412 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:40.517329 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:44.074649 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:44.074559 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:51.553601 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:51.553564 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:51.653365 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:51.653333 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:52.127284 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:52.127246 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:52.547601 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:52.547511 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:52.826258 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:52.826174 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:52.966482 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:52.966454 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:53.165525 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:53.165486 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:58.874429 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:58.874396 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:58.974473 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:58.974448 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:59.118698 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:59.118668 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:59.519460 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:59.519427 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:59.862779 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:59.862696 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:38:59.981321 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:38:59.981289 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:00.078377 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:00.078346 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:00.286206 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:00.286174 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:05.922861 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:05.922819 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:06.023209 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:06.023171 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:06.186380 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:06.186303 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:06.852176 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:06.852144 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:07.088653 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:07.088624 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:07.356600 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:07.356567 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:09.734640 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:09.734604 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:09.829634 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:09.829600 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:09.936338 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:09.936304 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:10.059249 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:10.059166 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:10.429419 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:10.429387 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:10.527892 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:10.527868 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:14.207634 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.207603 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:14.543201 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.543116 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:14.813020 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.812902 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc"] Jul 08 03:39:14.813335 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.813322 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0e2ec6b-d0b7-4366-811a-85e157284b45" containerName="main" Jul 08 03:39:14.813376 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.813336 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0e2ec6b-d0b7-4366-811a-85e157284b45" containerName="main" Jul 08 03:39:14.813376 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.813346 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1979fbee-3299-4cee-a183-71d4a9cfec7d" containerName="istio-proxy" Jul 08 03:39:14.813376 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.813352 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1979fbee-3299-4cee-a183-71d4a9cfec7d" containerName="istio-proxy" Jul 08 03:39:14.813376 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.813366 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b682100d-7267-4b2d-acef-bb78f37e1573" containerName="main" Jul 08 03:39:14.813376 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.813371 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b682100d-7267-4b2d-acef-bb78f37e1573" containerName="main" Jul 08 03:39:14.813525 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.813380 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1979fbee-3299-4cee-a183-71d4a9cfec7d" containerName="istio-proxy" Jul 08 03:39:14.813525 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.813385 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1979fbee-3299-4cee-a183-71d4a9cfec7d" containerName="istio-proxy" Jul 08 03:39:14.813525 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.813440 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b682100d-7267-4b2d-acef-bb78f37e1573" containerName="main" Jul 08 03:39:14.813525 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.813448 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0e2ec6b-d0b7-4366-811a-85e157284b45" containerName="main" Jul 08 03:39:14.813525 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.813457 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1979fbee-3299-4cee-a183-71d4a9cfec7d" containerName="istio-proxy" Jul 08 03:39:14.813525 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.813463 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1979fbee-3299-4cee-a183-71d4a9cfec7d" containerName="istio-proxy" Jul 08 03:39:14.816483 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.816464 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:14.819257 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.819234 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-705492-openshift-default-dockercfg-bcphw\"" Jul 08 03:39:14.830712 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.830692 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc"] Jul 08 03:39:14.846080 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.846050 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:14.866450 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.866428 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istio-data\") pod \"e2e-shared-b-705492-openshift-default-66578f65cf-jczjc\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:14.866587 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.866458 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istiod-ca-cert\") pod \"e2e-shared-b-705492-openshift-default-66578f65cf-jczjc\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:14.866587 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.866477 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6xhg\" (UniqueName: \"kubernetes.io/projected/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-kube-api-access-h6xhg\") pod \"e2e-shared-b-705492-openshift-default-66578f65cf-jczjc\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:14.866587 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.866547 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-workload-socket\") pod \"e2e-shared-b-705492-openshift-default-66578f65cf-jczjc\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:14.866703 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.866628 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-workload-certs\") pod \"e2e-shared-b-705492-openshift-default-66578f65cf-jczjc\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:14.866703 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.866666 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istio-token\") pod \"e2e-shared-b-705492-openshift-default-66578f65cf-jczjc\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:14.866785 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.866705 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istio-envoy\") pod \"e2e-shared-b-705492-openshift-default-66578f65cf-jczjc\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:14.866785 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.866776 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-credential-socket\") pod \"e2e-shared-b-705492-openshift-default-66578f65cf-jczjc\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:14.866858 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.866827 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istio-podinfo\") pod \"e2e-shared-b-705492-openshift-default-66578f65cf-jczjc\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:14.967795 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.967762 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istio-token\") pod \"e2e-shared-b-705492-openshift-default-66578f65cf-jczjc\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:14.967997 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.967816 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istio-envoy\") pod \"e2e-shared-b-705492-openshift-default-66578f65cf-jczjc\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:14.967997 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.967858 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-credential-socket\") pod \"e2e-shared-b-705492-openshift-default-66578f65cf-jczjc\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:14.967997 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.967885 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istio-podinfo\") pod \"e2e-shared-b-705492-openshift-default-66578f65cf-jczjc\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:14.967997 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.967913 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istio-data\") pod \"e2e-shared-b-705492-openshift-default-66578f65cf-jczjc\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:14.967997 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.967960 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istiod-ca-cert\") pod \"e2e-shared-b-705492-openshift-default-66578f65cf-jczjc\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:14.967997 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.967987 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6xhg\" (UniqueName: \"kubernetes.io/projected/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-kube-api-access-h6xhg\") pod \"e2e-shared-b-705492-openshift-default-66578f65cf-jczjc\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:14.968308 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.968037 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-workload-socket\") pod \"e2e-shared-b-705492-openshift-default-66578f65cf-jczjc\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:14.968308 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.968085 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-workload-certs\") pod \"e2e-shared-b-705492-openshift-default-66578f65cf-jczjc\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:14.968503 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.968474 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-credential-socket\") pod \"e2e-shared-b-705492-openshift-default-66578f65cf-jczjc\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:14.968558 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.968506 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-workload-certs\") pod \"e2e-shared-b-705492-openshift-default-66578f65cf-jczjc\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:14.968596 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.968517 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istio-data\") pod \"e2e-shared-b-705492-openshift-default-66578f65cf-jczjc\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:14.968644 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.968621 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-workload-socket\") pod \"e2e-shared-b-705492-openshift-default-66578f65cf-jczjc\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:14.969175 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.969142 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istiod-ca-cert\") pod \"e2e-shared-b-705492-openshift-default-66578f65cf-jczjc\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:14.970483 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.970458 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istio-envoy\") pod \"e2e-shared-b-705492-openshift-default-66578f65cf-jczjc\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:14.971040 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.971019 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istio-podinfo\") pod \"e2e-shared-b-705492-openshift-default-66578f65cf-jczjc\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:14.981564 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.981538 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6xhg\" (UniqueName: \"kubernetes.io/projected/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-kube-api-access-h6xhg\") pod \"e2e-shared-b-705492-openshift-default-66578f65cf-jczjc\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:14.982278 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.982256 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istio-token\") pod \"e2e-shared-b-705492-openshift-default-66578f65cf-jczjc\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:14.987950 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:14.987909 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:15.126310 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:15.126288 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:15.268582 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:15.268552 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc"] Jul 08 03:39:15.270953 ip-10-0-128-70 kubenswrapper[2564]: W0708 03:39:15.270896 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b7a19a_d75d_42a6_a844_64dbc2bfd02d.slice/crio-4796866b320f415613a157c5269cf5e239abd137b12b0f02c434c81f901ce953 WatchSource:0}: Error finding container 4796866b320f415613a157c5269cf5e239abd137b12b0f02c434c81f901ce953: Status 404 returned error can't find the container with id 4796866b320f415613a157c5269cf5e239abd137b12b0f02c434c81f901ce953 Jul 08 03:39:15.274058 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:15.274013 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:39:15.274179 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:15.274090 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:39:15.274179 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:15.274124 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:39:16.084842 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:16.084808 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" event={"ID":"34b7a19a-d75d-42a6-a844-64dbc2bfd02d","Type":"ContainerStarted","Data":"169643bb925c2a2c2ac73cc47eaec793b0d6f9cda1aed2af0e60bd91ec8ee75d"} Jul 08 03:39:16.085016 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:16.084858 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" event={"ID":"34b7a19a-d75d-42a6-a844-64dbc2bfd02d","Type":"ContainerStarted","Data":"4796866b320f415613a157c5269cf5e239abd137b12b0f02c434c81f901ce953"} Jul 08 03:39:16.107645 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:16.107601 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" podStartSLOduration=2.107587245 podStartE2EDuration="2.107587245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 03:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:39:16.106125047 +0000 UTC m=+2430.763500918" watchObservedRunningTime="2026-07-08 03:39:16.107587245 +0000 UTC m=+2430.764963116" Jul 08 03:39:16.127463 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:16.127438 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:16.132163 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:16.132139 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:17.090534 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:17.090507 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:17.091532 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:17.091514 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:18.051678 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:18.051639 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:18.159634 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:18.159601 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:18.243225 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:18.243199 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:18.341630 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:18.341549 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:18.942404 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:18.942375 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:19.042215 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:19.042177 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:22.502358 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:22.502329 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:30.149359 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:30.149325 2564 generic.go:356] "Generic (PLEG): container finished" podID="34b7a19a-d75d-42a6-a844-64dbc2bfd02d" containerID="169643bb925c2a2c2ac73cc47eaec793b0d6f9cda1aed2af0e60bd91ec8ee75d" exitCode=0 Jul 08 03:39:30.149766 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:30.149397 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" event={"ID":"34b7a19a-d75d-42a6-a844-64dbc2bfd02d","Type":"ContainerDied","Data":"169643bb925c2a2c2ac73cc47eaec793b0d6f9cda1aed2af0e60bd91ec8ee75d"} Jul 08 03:39:30.149818 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:30.149804 2564 scope.go:117] "RemoveContainer" containerID="169643bb925c2a2c2ac73cc47eaec793b0d6f9cda1aed2af0e60bd91ec8ee75d" Jul 08 03:39:30.150847 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:30.150813 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:39:30.150912 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:30.150887 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:39:30.150969 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:30.150939 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 03:39:31.156179 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:31.156146 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" event={"ID":"34b7a19a-d75d-42a6-a844-64dbc2bfd02d","Type":"ContainerStarted","Data":"a86fe1c5a43fbea7467b736603fcdd7903ad5e0727e7354d2903795ae1cebea0"} Jul 08 03:39:32.090422 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:32.090391 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:32.126558 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:32.126528 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:32.127661 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:32.127639 2564 patch_prober.go:28] interesting pod/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 08 03:39:32.127730 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:32.127688 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" podUID="34b7a19a-d75d-42a6-a844-64dbc2bfd02d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 08 03:39:32.205054 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:32.205017 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:32.303343 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:32.303293 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:33.127387 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:33.127341 2564 patch_prober.go:28] interesting pod/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 08 03:39:33.127571 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:33.127423 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" podUID="34b7a19a-d75d-42a6-a844-64dbc2bfd02d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 08 03:39:34.127077 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:34.127030 2564 patch_prober.go:28] interesting pod/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 08 03:39:34.127451 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:34.127108 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" podUID="34b7a19a-d75d-42a6-a844-64dbc2bfd02d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 08 03:39:35.130511 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:35.130483 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:35.131454 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:35.131436 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:39:39.781051 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:39.781011 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:39:39.881856 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:39:39.881821 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:42:27.412139 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:27.412062 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:42:27.757943 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:27.757835 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:42:27.858969 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:27.858939 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:42:27.959099 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:27.959067 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:42:28.096419 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:28.096342 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:42:33.419497 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:33.419459 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:42:35.052716 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:35.052688 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:42:39.045704 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:39.045665 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:42:39.093511 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:39.093486 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:42:40.074245 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:40.074208 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:42:40.174777 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:40.174737 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:42:40.307440 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:40.307409 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc"] Jul 08 03:42:40.307618 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:40.307598 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" podUID="34b7a19a-d75d-42a6-a844-64dbc2bfd02d" containerName="istio-proxy" containerID="cri-o://a86fe1c5a43fbea7467b736603fcdd7903ad5e0727e7354d2903795ae1cebea0" gracePeriod=30 Jul 08 03:42:40.322869 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:40.322842 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:42:43.710431 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:43.710397 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:42:45.556777 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.556753 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:42:45.605774 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.605699 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6xhg\" (UniqueName: \"kubernetes.io/projected/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-kube-api-access-h6xhg\") pod \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " Jul 08 03:42:45.605774 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.605731 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istio-token\") pod \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " Jul 08 03:42:45.605774 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.605776 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-workload-certs\") pod \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " Jul 08 03:42:45.606084 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.605828 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istio-podinfo\") pod \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " Jul 08 03:42:45.606084 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.605848 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-credential-socket\") pod \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " Jul 08 03:42:45.606084 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.605888 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-workload-socket\") pod \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " Jul 08 03:42:45.606084 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.605950 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istio-data\") pod \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " Jul 08 03:42:45.606084 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.605980 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istio-envoy\") pod \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " Jul 08 03:42:45.606084 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.606006 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istiod-ca-cert\") pod \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\" (UID: \"34b7a19a-d75d-42a6-a844-64dbc2bfd02d\") " Jul 08 03:42:45.606493 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.606138 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "34b7a19a-d75d-42a6-a844-64dbc2bfd02d" (UID: "34b7a19a-d75d-42a6-a844-64dbc2bfd02d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:42:45.606493 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.606359 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-workload-certs\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:42:45.606654 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.606524 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "34b7a19a-d75d-42a6-a844-64dbc2bfd02d" (UID: "34b7a19a-d75d-42a6-a844-64dbc2bfd02d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:42:45.607015 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.606835 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "34b7a19a-d75d-42a6-a844-64dbc2bfd02d" (UID: "34b7a19a-d75d-42a6-a844-64dbc2bfd02d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 03:42:45.607015 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.606907 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "34b7a19a-d75d-42a6-a844-64dbc2bfd02d" (UID: "34b7a19a-d75d-42a6-a844-64dbc2bfd02d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:42:45.607324 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.607015 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istio-data" (OuterVolumeSpecName: "istio-data") pod "34b7a19a-d75d-42a6-a844-64dbc2bfd02d" (UID: "34b7a19a-d75d-42a6-a844-64dbc2bfd02d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:42:45.608434 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.608407 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "34b7a19a-d75d-42a6-a844-64dbc2bfd02d" (UID: "34b7a19a-d75d-42a6-a844-64dbc2bfd02d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 03:42:45.608603 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.608449 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-kube-api-access-h6xhg" (OuterVolumeSpecName: "kube-api-access-h6xhg") pod "34b7a19a-d75d-42a6-a844-64dbc2bfd02d" (UID: "34b7a19a-d75d-42a6-a844-64dbc2bfd02d"). InnerVolumeSpecName "kube-api-access-h6xhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:42:45.608898 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.608874 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "34b7a19a-d75d-42a6-a844-64dbc2bfd02d" (UID: "34b7a19a-d75d-42a6-a844-64dbc2bfd02d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 03:42:45.609100 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.609078 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istio-token" (OuterVolumeSpecName: "istio-token") pod "34b7a19a-d75d-42a6-a844-64dbc2bfd02d" (UID: "34b7a19a-d75d-42a6-a844-64dbc2bfd02d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 03:42:45.610890 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.610868 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-2hkzh_a8816a59-d5ac-41c4-b35a-c7a914e61ac7/manager/0.log" Jul 08 03:42:45.707087 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.707047 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istio-podinfo\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:42:45.707087 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.707080 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-credential-socket\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:42:45.707087 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.707093 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-workload-socket\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:42:45.707322 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.707105 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istio-data\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:42:45.707322 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.707116 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istio-envoy\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:42:45.707322 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.707127 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istiod-ca-cert\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:42:45.707322 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.707136 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6xhg\" (UniqueName: \"kubernetes.io/projected/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-kube-api-access-h6xhg\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:42:45.707322 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:45.707148 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b7a19a-d75d-42a6-a844-64dbc2bfd02d-istio-token\") on node \"ip-10-0-128-70.ec2.internal\" DevicePath \"\"" Jul 08 03:42:46.068060 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:46.068029 2564 generic.go:356] "Generic (PLEG): container finished" podID="34b7a19a-d75d-42a6-a844-64dbc2bfd02d" containerID="a86fe1c5a43fbea7467b736603fcdd7903ad5e0727e7354d2903795ae1cebea0" exitCode=0 Jul 08 03:42:46.068240 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:46.068076 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" Jul 08 03:42:46.068240 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:46.068084 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" event={"ID":"34b7a19a-d75d-42a6-a844-64dbc2bfd02d","Type":"ContainerDied","Data":"a86fe1c5a43fbea7467b736603fcdd7903ad5e0727e7354d2903795ae1cebea0"} Jul 08 03:42:46.068240 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:46.068128 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc" event={"ID":"34b7a19a-d75d-42a6-a844-64dbc2bfd02d","Type":"ContainerDied","Data":"4796866b320f415613a157c5269cf5e239abd137b12b0f02c434c81f901ce953"} Jul 08 03:42:46.068240 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:46.068149 2564 scope.go:117] "RemoveContainer" containerID="a86fe1c5a43fbea7467b736603fcdd7903ad5e0727e7354d2903795ae1cebea0" Jul 08 03:42:46.076865 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:46.076852 2564 scope.go:117] "RemoveContainer" containerID="169643bb925c2a2c2ac73cc47eaec793b0d6f9cda1aed2af0e60bd91ec8ee75d" Jul 08 03:42:46.085019 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:46.084997 2564 scope.go:117] "RemoveContainer" containerID="a86fe1c5a43fbea7467b736603fcdd7903ad5e0727e7354d2903795ae1cebea0" Jul 08 03:42:46.085151 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:46.085131 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc"] Jul 08 03:42:46.085283 ip-10-0-128-70 kubenswrapper[2564]: E0708 03:42:46.085265 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a86fe1c5a43fbea7467b736603fcdd7903ad5e0727e7354d2903795ae1cebea0\": container with ID starting with a86fe1c5a43fbea7467b736603fcdd7903ad5e0727e7354d2903795ae1cebea0 not found: ID does not exist" containerID="a86fe1c5a43fbea7467b736603fcdd7903ad5e0727e7354d2903795ae1cebea0" Jul 08 03:42:46.085329 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:46.085292 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a86fe1c5a43fbea7467b736603fcdd7903ad5e0727e7354d2903795ae1cebea0"} err="failed to get container status \"a86fe1c5a43fbea7467b736603fcdd7903ad5e0727e7354d2903795ae1cebea0\": rpc error: code = NotFound desc = could not find container \"a86fe1c5a43fbea7467b736603fcdd7903ad5e0727e7354d2903795ae1cebea0\": container with ID starting with a86fe1c5a43fbea7467b736603fcdd7903ad5e0727e7354d2903795ae1cebea0 not found: ID does not exist" Jul 08 03:42:46.085329 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:46.085308 2564 scope.go:117] "RemoveContainer" containerID="169643bb925c2a2c2ac73cc47eaec793b0d6f9cda1aed2af0e60bd91ec8ee75d" Jul 08 03:42:46.085656 ip-10-0-128-70 kubenswrapper[2564]: E0708 03:42:46.085631 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"169643bb925c2a2c2ac73cc47eaec793b0d6f9cda1aed2af0e60bd91ec8ee75d\": container with ID starting with 169643bb925c2a2c2ac73cc47eaec793b0d6f9cda1aed2af0e60bd91ec8ee75d not found: ID does not exist" containerID="169643bb925c2a2c2ac73cc47eaec793b0d6f9cda1aed2af0e60bd91ec8ee75d" Jul 08 03:42:46.085781 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:46.085664 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"169643bb925c2a2c2ac73cc47eaec793b0d6f9cda1aed2af0e60bd91ec8ee75d"} err="failed to get container status \"169643bb925c2a2c2ac73cc47eaec793b0d6f9cda1aed2af0e60bd91ec8ee75d\": rpc error: code = NotFound desc = could not find container \"169643bb925c2a2c2ac73cc47eaec793b0d6f9cda1aed2af0e60bd91ec8ee75d\": container with ID starting with 169643bb925c2a2c2ac73cc47eaec793b0d6f9cda1aed2af0e60bd91ec8ee75d not found: ID does not exist" Jul 08 03:42:46.088715 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:46.088693 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-705492-openshift-default-66578f65cf-jczjc"] Jul 08 03:42:46.318735 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:46.318660 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64b8ffddd6-cfk9b_4d827dbf-0236-4f9c-b93b-5ed89ada804f/copy-manifests/0.log" Jul 08 03:42:46.324931 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:46.324904 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64b8ffddd6-cfk9b_4d827dbf-0236-4f9c-b93b-5ed89ada804f/manager/0.log" Jul 08 03:42:47.886163 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:47.886132 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34b7a19a-d75d-42a6-a844-64dbc2bfd02d" path="/var/lib/kubelet/pods/34b7a19a-d75d-42a6-a844-64dbc2bfd02d/volumes" Jul 08 03:42:47.911643 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:47.911616 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-p7fjt_b5f5ba59-ab30-4609-9c91-4e205ee9710b/kuadrant-console-plugin/0.log" Jul 08 03:42:48.707059 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:48.707034 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4vhx4r_059975b3-ece8-476b-a559-45b6d1c2bf95/istio-proxy/0.log" Jul 08 03:42:48.928186 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:48.928156 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58b85c4fb4-cvslw_076a45d6-5c31-4e82-ac94-84aafddcdf69/kube-auth-proxy/0.log" Jul 08 03:42:49.152452 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:49.152424 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lzlg7_631c38ff-a249-470e-b1c5-bf4f761e4b71/istio-proxy/1.log" Jul 08 03:42:50.147651 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:50.147617 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkvqg8_3af5c35f-19c9-4917-84d1-a5c432424f0f/main/0.log" Jul 08 03:42:50.257280 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:50.257247 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-2q7sw_36f04153-1335-43f6-b684-f30e05c3e76c/main/0.log" Jul 08 03:42:50.360102 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:50.360068 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-n9qhz_e35a160f-a467-4c57-b6e3-ec79232b3b23/main/0.log" Jul 08 03:42:54.201896 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:54.201864 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:42:54.299177 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:54.299143 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" limit=1024 Jul 08 03:42:57.075441 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:57.075405 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-54p5l_a6e66819-f708-4e22-81d4-46adeef611e0/global-pull-secret-syncer/0.log" Jul 08 03:42:57.200088 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:57.200053 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4ccc2_6c992baa-d9c2-4241-9c52-9c930569f591/konnectivity-agent/0.log" Jul 08 03:42:57.280404 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:42:57.280374 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-70.ec2.internal_b68f5c5921db369bc5547aa00eae2349/haproxy/0.log" Jul 08 03:43:02.460446 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:02.460416 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-p7fjt_b5f5ba59-ab30-4609-9c91-4e205ee9710b/kuadrant-console-plugin/0.log" Jul 08 03:43:04.258151 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:04.258126 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_878d90d4-98a5-4075-a5ec-3f20b859838a/alertmanager/0.log" Jul 08 03:43:04.276296 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:04.276275 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_878d90d4-98a5-4075-a5ec-3f20b859838a/config-reloader/0.log" Jul 08 03:43:04.294911 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:04.294892 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_878d90d4-98a5-4075-a5ec-3f20b859838a/kube-rbac-proxy-web/0.log" Jul 08 03:43:04.315402 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:04.315381 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_878d90d4-98a5-4075-a5ec-3f20b859838a/kube-rbac-proxy/0.log" Jul 08 03:43:04.335899 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:04.335876 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_878d90d4-98a5-4075-a5ec-3f20b859838a/kube-rbac-proxy-metric/0.log" Jul 08 03:43:04.352620 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:04.352599 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_878d90d4-98a5-4075-a5ec-3f20b859838a/prom-label-proxy/0.log" Jul 08 03:43:04.370354 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:04.370334 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_878d90d4-98a5-4075-a5ec-3f20b859838a/init-config-reloader/0.log" Jul 08 03:43:04.474125 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:04.474042 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-zz8c7_94fa01b1-87d6-41db-ad69-938f57cd77fe/kube-state-metrics/0.log" Jul 08 03:43:04.495302 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:04.495283 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-zz8c7_94fa01b1-87d6-41db-ad69-938f57cd77fe/kube-rbac-proxy-main/0.log" Jul 08 03:43:04.517304 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:04.517285 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-zz8c7_94fa01b1-87d6-41db-ad69-938f57cd77fe/kube-rbac-proxy-self/0.log" Jul 08 03:43:04.796596 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:04.796528 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v2tk7_3c1a5c79-6201-494e-b04f-a3c4dbc7c90c/node-exporter/0.log" Jul 08 03:43:04.817546 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:04.817524 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v2tk7_3c1a5c79-6201-494e-b04f-a3c4dbc7c90c/kube-rbac-proxy/0.log" Jul 08 03:43:04.840611 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:04.840586 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v2tk7_3c1a5c79-6201-494e-b04f-a3c4dbc7c90c/init-textfile/0.log" Jul 08 03:43:04.970689 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:04.970659 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0ba3abfa-0a04-476f-8354-0f02bd55bf08/prometheus/0.log" Jul 08 03:43:04.992121 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:04.992092 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0ba3abfa-0a04-476f-8354-0f02bd55bf08/config-reloader/0.log" Jul 08 03:43:05.012466 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:05.012434 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0ba3abfa-0a04-476f-8354-0f02bd55bf08/thanos-sidecar/0.log" Jul 08 03:43:05.034637 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:05.034617 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0ba3abfa-0a04-476f-8354-0f02bd55bf08/kube-rbac-proxy-web/0.log" Jul 08 03:43:05.061365 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:05.061273 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0ba3abfa-0a04-476f-8354-0f02bd55bf08/kube-rbac-proxy/0.log" Jul 08 03:43:05.075471 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:05.075445 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0ba3abfa-0a04-476f-8354-0f02bd55bf08/kube-rbac-proxy-thanos/0.log" Jul 08 03:43:05.098232 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:05.098215 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0ba3abfa-0a04-476f-8354-0f02bd55bf08/init-config-reloader/0.log" Jul 08 03:43:05.123330 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:05.123293 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-74t5g_e8f8dbad-7d14-4946-a94e-507e7405141b/prometheus-operator/0.log" Jul 08 03:43:05.142765 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:05.142695 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-74t5g_e8f8dbad-7d14-4946-a94e-507e7405141b/kube-rbac-proxy/0.log" Jul 08 03:43:05.267879 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:05.267851 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-575dbdcf76-mt7x6_2701c118-eb37-4c11-917c-010285e54cb7/thanos-query/0.log" Jul 08 03:43:05.285065 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:05.285025 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-575dbdcf76-mt7x6_2701c118-eb37-4c11-917c-010285e54cb7/kube-rbac-proxy-web/0.log" Jul 08 03:43:05.308748 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:05.308729 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-575dbdcf76-mt7x6_2701c118-eb37-4c11-917c-010285e54cb7/kube-rbac-proxy/0.log" Jul 08 03:43:05.326183 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:05.326120 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-575dbdcf76-mt7x6_2701c118-eb37-4c11-917c-010285e54cb7/prom-label-proxy/0.log" Jul 08 03:43:05.347351 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:05.347301 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-575dbdcf76-mt7x6_2701c118-eb37-4c11-917c-010285e54cb7/kube-rbac-proxy-rules/0.log" Jul 08 03:43:05.365977 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:05.365954 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-575dbdcf76-mt7x6_2701c118-eb37-4c11-917c-010285e54cb7/kube-rbac-proxy-metrics/0.log" Jul 08 03:43:07.095032 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.094994 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-l26gf/perf-node-gather-daemonset-nvnl7"] Jul 08 03:43:07.095403 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.095337 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34b7a19a-d75d-42a6-a844-64dbc2bfd02d" containerName="istio-proxy" Jul 08 03:43:07.095403 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.095348 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="34b7a19a-d75d-42a6-a844-64dbc2bfd02d" containerName="istio-proxy" Jul 08 03:43:07.095403 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.095356 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34b7a19a-d75d-42a6-a844-64dbc2bfd02d" containerName="istio-proxy" Jul 08 03:43:07.095403 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.095362 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="34b7a19a-d75d-42a6-a844-64dbc2bfd02d" containerName="istio-proxy" Jul 08 03:43:07.095535 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.095436 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="34b7a19a-d75d-42a6-a844-64dbc2bfd02d" containerName="istio-proxy" Jul 08 03:43:07.095535 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.095446 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="34b7a19a-d75d-42a6-a844-64dbc2bfd02d" containerName="istio-proxy" Jul 08 03:43:07.098355 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.098334 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-nvnl7" Jul 08 03:43:07.101015 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.100993 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l26gf\"/\"kube-root-ca.crt\"" Jul 08 03:43:07.101107 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.100996 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l26gf\"/\"openshift-service-ca.crt\"" Jul 08 03:43:07.102055 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.102041 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-l26gf\"/\"default-dockercfg-x5rmv\"" Jul 08 03:43:07.109133 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.109113 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l26gf/perf-node-gather-daemonset-nvnl7"] Jul 08 03:43:07.193882 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.193849 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvkrt\" (UniqueName: \"kubernetes.io/projected/c6810a22-0da7-4d1e-8939-1f8d221f33ba-kube-api-access-nvkrt\") pod \"perf-node-gather-daemonset-nvnl7\" (UID: \"c6810a22-0da7-4d1e-8939-1f8d221f33ba\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-nvnl7" Jul 08 03:43:07.193882 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.193883 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6810a22-0da7-4d1e-8939-1f8d221f33ba-lib-modules\") pod \"perf-node-gather-daemonset-nvnl7\" (UID: \"c6810a22-0da7-4d1e-8939-1f8d221f33ba\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-nvnl7" Jul 08 03:43:07.194102 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.193905 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c6810a22-0da7-4d1e-8939-1f8d221f33ba-proc\") pod \"perf-node-gather-daemonset-nvnl7\" (UID: \"c6810a22-0da7-4d1e-8939-1f8d221f33ba\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-nvnl7" Jul 08 03:43:07.194102 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.194045 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6810a22-0da7-4d1e-8939-1f8d221f33ba-sys\") pod \"perf-node-gather-daemonset-nvnl7\" (UID: \"c6810a22-0da7-4d1e-8939-1f8d221f33ba\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-nvnl7" Jul 08 03:43:07.194102 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.194082 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c6810a22-0da7-4d1e-8939-1f8d221f33ba-podres\") pod \"perf-node-gather-daemonset-nvnl7\" (UID: \"c6810a22-0da7-4d1e-8939-1f8d221f33ba\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-nvnl7" Jul 08 03:43:07.295416 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.295380 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvkrt\" (UniqueName: \"kubernetes.io/projected/c6810a22-0da7-4d1e-8939-1f8d221f33ba-kube-api-access-nvkrt\") pod \"perf-node-gather-daemonset-nvnl7\" (UID: \"c6810a22-0da7-4d1e-8939-1f8d221f33ba\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-nvnl7" Jul 08 03:43:07.295416 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.295419 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6810a22-0da7-4d1e-8939-1f8d221f33ba-lib-modules\") pod \"perf-node-gather-daemonset-nvnl7\" (UID: \"c6810a22-0da7-4d1e-8939-1f8d221f33ba\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-nvnl7" Jul 08 03:43:07.295612 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.295435 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c6810a22-0da7-4d1e-8939-1f8d221f33ba-proc\") pod \"perf-node-gather-daemonset-nvnl7\" (UID: \"c6810a22-0da7-4d1e-8939-1f8d221f33ba\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-nvnl7" Jul 08 03:43:07.295612 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.295497 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6810a22-0da7-4d1e-8939-1f8d221f33ba-sys\") pod \"perf-node-gather-daemonset-nvnl7\" (UID: \"c6810a22-0da7-4d1e-8939-1f8d221f33ba\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-nvnl7" Jul 08 03:43:07.295612 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.295527 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c6810a22-0da7-4d1e-8939-1f8d221f33ba-podres\") pod \"perf-node-gather-daemonset-nvnl7\" (UID: \"c6810a22-0da7-4d1e-8939-1f8d221f33ba\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-nvnl7" Jul 08 03:43:07.295612 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.295584 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6810a22-0da7-4d1e-8939-1f8d221f33ba-lib-modules\") pod \"perf-node-gather-daemonset-nvnl7\" (UID: \"c6810a22-0da7-4d1e-8939-1f8d221f33ba\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-nvnl7" Jul 08 03:43:07.295612 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.295590 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c6810a22-0da7-4d1e-8939-1f8d221f33ba-proc\") pod \"perf-node-gather-daemonset-nvnl7\" (UID: \"c6810a22-0da7-4d1e-8939-1f8d221f33ba\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-nvnl7" Jul 08 03:43:07.295612 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.295601 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6810a22-0da7-4d1e-8939-1f8d221f33ba-sys\") pod \"perf-node-gather-daemonset-nvnl7\" (UID: \"c6810a22-0da7-4d1e-8939-1f8d221f33ba\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-nvnl7" Jul 08 03:43:07.295810 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.295696 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c6810a22-0da7-4d1e-8939-1f8d221f33ba-podres\") pod \"perf-node-gather-daemonset-nvnl7\" (UID: \"c6810a22-0da7-4d1e-8939-1f8d221f33ba\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-nvnl7" Jul 08 03:43:07.304803 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.304782 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvkrt\" (UniqueName: \"kubernetes.io/projected/c6810a22-0da7-4d1e-8939-1f8d221f33ba-kube-api-access-nvkrt\") pod \"perf-node-gather-daemonset-nvnl7\" (UID: \"c6810a22-0da7-4d1e-8939-1f8d221f33ba\") " pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-nvnl7" Jul 08 03:43:07.409107 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.409080 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-nvnl7" Jul 08 03:43:07.537295 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:07.537264 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l26gf/perf-node-gather-daemonset-nvnl7"] Jul 08 03:43:07.540341 ip-10-0-128-70 kubenswrapper[2564]: W0708 03:43:07.540317 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc6810a22_0da7_4d1e_8939_1f8d221f33ba.slice/crio-01ef00003b8d3347a21b8639ffc993347c7b5f7d01544f30f5af7807ca130755 WatchSource:0}: Error finding container 01ef00003b8d3347a21b8639ffc993347c7b5f7d01544f30f5af7807ca130755: Status 404 returned error can't find the container with id 01ef00003b8d3347a21b8639ffc993347c7b5f7d01544f30f5af7807ca130755 Jul 08 03:43:08.178041 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:08.178003 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-nvnl7" event={"ID":"c6810a22-0da7-4d1e-8939-1f8d221f33ba","Type":"ContainerStarted","Data":"ce54f0c9947fc45b82c70fbf8be6f65dbbf24d8d3c93fba1bdb69c30c528818b"} Jul 08 03:43:08.178453 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:08.178064 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-nvnl7" event={"ID":"c6810a22-0da7-4d1e-8939-1f8d221f33ba","Type":"ContainerStarted","Data":"01ef00003b8d3347a21b8639ffc993347c7b5f7d01544f30f5af7807ca130755"} Jul 08 03:43:08.178453 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:08.178132 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-nvnl7" Jul 08 03:43:08.195378 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:08.195332 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-nvnl7" podStartSLOduration=1.195319149 podStartE2EDuration="1.195319149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 03:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 03:43:08.194558422 +0000 UTC m=+2662.851934293" watchObservedRunningTime="2026-07-08 03:43:08.195319149 +0000 UTC m=+2662.852695073" Jul 08 03:43:09.584695 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:09.584659 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rkrlf_cd436540-823b-4c37-9a20-b8ebdf0937b7/dns/0.log" Jul 08 03:43:09.603868 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:09.603847 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rkrlf_cd436540-823b-4c37-9a20-b8ebdf0937b7/kube-rbac-proxy/0.log" Jul 08 03:43:09.626242 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:09.626220 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8jg82_2d241eb6-7dbd-4ad7-86ef-6ba3fd7bf61c/dns-node-resolver/0.log" Jul 08 03:43:10.196049 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:10.196023 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7v4gn_97c25edf-e111-4f3c-933f-9e45c6926b2c/node-ca/0.log" Jul 08 03:43:11.123582 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:11.123549 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4vhx4r_059975b3-ece8-476b-a559-45b6d1c2bf95/istio-proxy/0.log" Jul 08 03:43:11.261874 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:11.261846 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58b85c4fb4-cvslw_076a45d6-5c31-4e82-ac94-84aafddcdf69/kube-auth-proxy/0.log" Jul 08 03:43:11.313954 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:11.313904 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lzlg7_631c38ff-a249-470e-b1c5-bf4f761e4b71/istio-proxy/0.log" Jul 08 03:43:11.339688 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:11.339664 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lzlg7_631c38ff-a249-470e-b1c5-bf4f761e4b71/istio-proxy/1.log" Jul 08 03:43:12.146211 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:12.146176 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b49sr_9489f6b7-c200-4e47-93f7-2f8a8e4be132/serve-healthcheck-canary/0.log" Jul 08 03:43:12.733556 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:12.733526 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ndv6v_4bd7623c-f229-43b7-aefd-339ec594cf2c/kube-rbac-proxy/0.log" Jul 08 03:43:12.753527 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:12.753506 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ndv6v_4bd7623c-f229-43b7-aefd-339ec594cf2c/exporter/0.log" Jul 08 03:43:12.772079 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:12.772056 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ndv6v_4bd7623c-f229-43b7-aefd-339ec594cf2c/extractor/0.log" Jul 08 03:43:14.191671 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:14.191644 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-l26gf/perf-node-gather-daemonset-nvnl7" Jul 08 03:43:15.036401 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:15.036370 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-2hkzh_a8816a59-d5ac-41c4-b35a-c7a914e61ac7/manager/0.log" Jul 08 03:43:15.397220 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:15.397194 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64b8ffddd6-cfk9b_4d827dbf-0236-4f9c-b93b-5ed89ada804f/manager/0.log" Jul 08 03:43:15.415587 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:15.415565 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64b8ffddd6-cfk9b_4d827dbf-0236-4f9c-b93b-5ed89ada804f/copy-manifests/0.log" Jul 08 03:43:16.866450 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:16.866385 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-65cfbc7dbc-cwswd_b965c3f1-ed80-4b1f-8448-3df5096345e0/manager/0.log" Jul 08 03:43:16.917648 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:16.917625 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-mwcwh_9bf4a7d2-fa7c-4447-b950-b03527ca9d3e/openshift-lws-operator/0.log" Jul 08 03:43:17.805595 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:17.805564 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4vhx4r_059975b3-ece8-476b-a559-45b6d1c2bf95/istio-proxy/0.log" Jul 08 03:43:17.913659 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:17.913634 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58b85c4fb4-cvslw_076a45d6-5c31-4e82-ac94-84aafddcdf69/kube-auth-proxy/0.log" Jul 08 03:43:17.951602 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:17.951574 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lzlg7_631c38ff-a249-470e-b1c5-bf4f761e4b71/istio-proxy/0.log" Jul 08 03:43:17.978549 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:17.978526 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lzlg7_631c38ff-a249-470e-b1c5-bf4f761e4b71/istio-proxy/1.log" Jul 08 03:43:19.280744 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:19.280718 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4vhx4r_059975b3-ece8-476b-a559-45b6d1c2bf95/istio-proxy/0.log" Jul 08 03:43:19.401774 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:19.401749 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58b85c4fb4-cvslw_076a45d6-5c31-4e82-ac94-84aafddcdf69/kube-auth-proxy/0.log" Jul 08 03:43:19.436064 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:19.436038 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lzlg7_631c38ff-a249-470e-b1c5-bf4f761e4b71/istio-proxy/0.log" Jul 08 03:43:19.465725 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:19.465694 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lzlg7_631c38ff-a249-470e-b1c5-bf4f761e4b71/istio-proxy/1.log" Jul 08 03:43:23.421312 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:23.421283 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lwl2s_fce35c6b-7b91-4a96-8f3f-4227c1ef6127/kube-multus-additional-cni-plugins/0.log" Jul 08 03:43:23.438686 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:23.438662 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lwl2s_fce35c6b-7b91-4a96-8f3f-4227c1ef6127/egress-router-binary-copy/0.log" Jul 08 03:43:23.455600 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:23.455576 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lwl2s_fce35c6b-7b91-4a96-8f3f-4227c1ef6127/cni-plugins/0.log" Jul 08 03:43:23.472829 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:23.472809 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lwl2s_fce35c6b-7b91-4a96-8f3f-4227c1ef6127/bond-cni-plugin/0.log" Jul 08 03:43:23.492855 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:23.492830 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lwl2s_fce35c6b-7b91-4a96-8f3f-4227c1ef6127/routeoverride-cni/0.log" Jul 08 03:43:23.510188 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:23.510166 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lwl2s_fce35c6b-7b91-4a96-8f3f-4227c1ef6127/whereabouts-cni-bincopy/0.log" Jul 08 03:43:23.528825 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:23.528806 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lwl2s_fce35c6b-7b91-4a96-8f3f-4227c1ef6127/whereabouts-cni/0.log" Jul 08 03:43:23.690430 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:23.690375 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-b25jn_570103f4-bd12-4dd7-813a-7d3a2028ff68/kube-multus/0.log" Jul 08 03:43:23.764738 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:23.764715 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lsmbz_8f5283b1-208f-415c-819f-c3e479102863/network-metrics-daemon/0.log" Jul 08 03:43:23.782280 ip-10-0-128-70 kubenswrapper[2564]: I0708 03:43:23.782260 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lsmbz_8f5283b1-208f-415c-819f-c3e479102863/kube-rbac-proxy/0.log"